VOOZH about

URL: https://arxiv.org/abs/2502.18525

⇱ [2502.18525] Programming with Pixels: Can Computer-Use Agents do Software Engineering?

Computer Science > Software Engineering

arXiv:2502.18525 (cs)
[Submitted on 24 Feb 2025 (v1), last revised 3 Oct 2025 (this version, v2)]

Title:Programming with Pixels: Can Computer-Use Agents do Software Engineering?

View PDF HTML (experimental)
Abstract:Computer-use agents (CUAs) hold the promise of performing a wide variety of general tasks, but current evaluations have primarily focused on simple scenarios. It therefore remains unclear whether such generalist agents can automate more sophisticated and specialized work such as software engineering (SWE). To investigate this, we introduce $\texttt{Programming with Pixels}$ (PwP), the first comprehensive computer-use environment for software engineering, where agents visually control an IDE to perform diverse software engineering tasks. To enable holistic evaluation, we also introduce \texttt{PwP-Bench}, a benchmark of 15 existing and new software-engineering tasks spanning multiple modalities, programming languages, and skillsets. We perform an extensive evaluation of state-of-the-art open-weight and closed-weight CUAs and find that when interacting purely visually, they perform significantly worse than specialized coding agents. However, when the same CUAs are given direct access to just two APIs-file editing and bash operations-performance jumps, often reaching the levels of specialized agents despite having a task-agnostic design. Furthermore, when given access to additional IDE tools via text APIs, all models show further gains. Our analysis shows that current CUAs fall short mainly due to limited visual grounding and the inability to take full advantage of the rich environment, leaving clear room for future this http URL establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance on sophisticated tasks. Code and data released at this https URL
Subjects: Software Engineering (cs.SE); Machine Learning (cs.LG)
Cite as: arXiv:2502.18525 [cs.SE]
(or arXiv:2502.18525v2 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2502.18525
arXiv-issued DOI via DataCite

Submission history

From: Pranjal Aggarwal [view email]
[v1] Mon, 24 Feb 2025 18:41:33 UTC (5,875 KB)
[v2] Fri, 3 Oct 2025 02:09:59 UTC (5,293 KB)
Full-text links:

Access Paper:

👁 license icon
view license

Current browse context:

cs.SE
< prev   |   next >
Change to browse by:

References & Citations

BibTeX formatted citation

Data provided by:

Bibliographic and Citation Tools

Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)

Code, Data and Media Associated with this Article

alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)

Demos

Replicate (What is Replicate?)
Hugging Face Spaces (What is Spaces?)
TXYZ.AI (What is TXYZ.AI?)

Recommenders and Search Tools

Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.