Have you looked into METR at all? They’ve done some interesting work on measuring AI model capability and autonomy against a benchmark of software engineering tasks which have baseline measurements of how long they take a human to do.
TL;DR, frontier models currently have a ~50% success rate when performing tasks that would take humans a little over two hours, and that time has been doubling every 7 months. Assuming that trajectory, they’ll have the autonomy to complete a human 40 hour work week worth of work in less than three years.
https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
Doc Orange / Tommy Volk
npub1jg…kqcgk
2025-11-04 07:42:20 UTC
in reply to nevent1q…a04t
Author Public Key
npub1jgnwmgfxcch992rkaqm5sk740vzsnpnmkrrnrm3pxu3uv6smmc8qhkqcgkSeen on
wss://relay.bitcoinpark.comPublished at
2025-11-04 07:42:20 UTCEvent JSON
{
"id": "25c87e3578c5c0d04ba6a32384b9dc938d9c2e2456b6e89cbb778b61a73851c1",
"pubkey": "9226eda126c62e52a876e837485bd57b0509867bb0c731ee213723c66a1bde0e",
"created_at": 1762242140,
"kind": 1,
"tags": [
[
"e",
"b4ee3d0c65f644f7a6abd9324a7e0394b582494321897486fab1114dd2e1088c",
"wss://relay.primal.net/",
"root",
"99bb5591c9116600f845107d31f9b59e2f7c7e09a1ff802e84f1d43da557ca64"
],
[
"e",
"0d8b932d009495b6385f085f8be4937447bfaad840794475483c3a4fe2217914",
"wss://relay.primal.net",
"reply"
],
[
"p",
"553eb2cad2f273d07e378eebd94be9abf984ce45963b7309899477333a135132",
"",
"mention"
],
[
"p",
"99bb5591c9116600f845107d31f9b59e2f7c7e09a1ff802e84f1d43da557ca64",
"",
"mention"
]
],
"content": "Have you looked into METR at all? They’ve done some interesting work on measuring AI model capability and autonomy against a benchmark of software engineering tasks which have baseline measurements of how long they take a human to do.\n\nTL;DR, frontier models currently have a ~50% success rate when performing tasks that would take humans a little over two hours, and that time has been doubling every 7 months. Assuming that trajectory, they’ll have the autonomy to complete a human 40 hour work week worth of work in less than three years.\n\nhttps://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/",
"sig": "9b948741990c4cb3b8b19c54e66ef824f5d4fa89e1c1ddc1f0847b067ab7ea6705eaabd48cb897f1dda99af03114776e3607d2342f87e61465a493e301ad8b13"
}