Newshot

Zero-Dimensional Attention Topology for Linguistic Acceptability

Summary

arXiv:2611.22312v1 Announce Type: new Abstract: Topological summaries of Transformer attention provide sentence representations for linguistic acceptability classification. We examine a zero-dimensional persistent-homology ($H_0$) representation that summarizes minimum-spanning-tree merge values from 144 symmetrized BERT attention graphs into 1,440 features. The retrospective evaluation uses 8,517 downstream training sentences and 521 in-domain and 497 out-of-domain CoLA validation sentences after data-integrity filtering. The recorded out-of-domain Matthews correlation coefficients are 0.438 for the topological representation and 0.508 for attention statistics, compared with 0.545 for BERT. Attention-feature results are retained aggregate measurements; BERT predictions are rescored and length controls are refitted from the accompanying records. Analytical results show that the representation retains token-count information and discards a class of directional attention differences. Sou…

Original text(via arXiv cs.CL)

Computer Science > Computation and Language

Title:Zero-Dimensional Attention Topology for Linguistic Acceptability

Abstract:Topological summaries of Transformer attention provide sentence representations for linguistic acceptability classification. We examine a zero-dimensional persistent-homology ($H_0$) representation that summarizes minimum-spanning-tree merge values from 144 symmetrized BERT attention graphs into 1,440 features. The retrospective evaluation uses 8,517 downstream training sentences and 521 in-domain and 497 out-of-domain CoLA validation sentences after data-integrity filtering. The recorded out-of-domain Matthews correlation coefficients are 0.438 for the topological representation and 0.508 for attention statistics, compared with 0.545 for BERT. Attention-feature results are retained aggregate measurements; BERT predictions are rescored and length controls are refitted from the accompanying records. Analytical results show that the representation retains token-count information and discards a class of directional attention differences. Source resampling of BERT predictions also demonstrates the sensitivity of uncertainty estimates to the sampling unit. The reported results do not demonstrate a predictive advantage for this fixed topological representation over simpler attention summaries. They identify representation properties and evaluation controls relevant to subsequent studies of attention topology.
Comments:10 pages, 4 figures, 4 tables. Code and data are provided as ancillary files
Subjects:Computation and Language (cs.CL) ; Machine Learning (cs.LG)
Cite as:arXiv:2611.22312 [cs.CL]
(or arXiv:2611.22312v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.22312 arXiv-issued DOI via DataCite (pending registration)

Submission history

Access Paper:

  • HTML (experimental)

Ancillary files (details):

  • audit_rows.csv
  • audit_summary.json
  • baseline_predictions.csv
  • bert_source_metrics.csv
  • build_audit.py
  • cluster_sensitivity.json
  • execution_environment.json
  • h0_features.py
  • historical_contrasts.csv
  • historical_results.csv
  • input_manifest.json
  • length_models.json
  • make_figures.py
  • metric_verification.json
  • reproduce_results.py
  • requirements.txt
  • source_provenance.json
  • synthetic_tests.json
  • verified_metrics.csv
  • verify_metrics.py
  • (17 additional files not shown)

Additional Features

  • Audio Summary

Current browse context:

References & Citations

  • Google Scholar
  • Semantic Scholar

BibTeX formatted citation

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by arXiv cs.CL
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗