When an AI system encounters incomplete, inconsistent, or contradictory information, how should it determine whether there is enough evidence to produce a reliable answer?

This is an important problem in natural language processing, information extraction, and AI-assisted decision-making. A system may retrieve information from multiple sources, encounter missing values, or find conflicting claims that cannot be resolved using the available evidence.

One possible approach is to use evidence-first, fail-closed decision rules. Under this approach, a system would not proceed with scoring or evaluation when essential evidence or required conditions remain unresolved. Missing information would remain explicitly missing rather than being silently replaced with assumed values.

I am exploring these principles through the Organic Intelligence Protocol (OIP), a research-stage framework focused on evidence requirements, auditability, and claim discipline. The framework has not been independently validated as a general machine learning method, and its documented audits do not establish predictive superiority or verified real-world AI safety.

I would appreciate technical perspectives on three questions.

1. How can evidence sufficiency be formally defined and measured in an NLP pipeline?

2. What evaluation methods could determine whether fail-closed rules reduce unsupported conclusions without unnecessarily rejecting valid answers?

3. How should a system distinguish missing information, contradictory evidence, and genuine uncertainty?

I am particularly interested in approaches involving information extraction, knowledge representation, retrieval-augmented generation, and uncertainty evaluation.

What existing methods or experimental designs would be most appropriate for investigating these questions?

Did anyone get papers to review for the ARR October cycle?

Update: They assigned 10 Papers, damn.

Context: I come from a linguistics background and have several years of experience in my specialty within the field. I passed the first round of interviews for an EdTech learning company for a high-ish level role. They gave me an "assignment" to see how I would handle what I assume are issues they are facing regarding scoring, mapping of scores, comparisons, etc.

The assignment has multiple parts and I have been explicitly asked for all of my thinking regarding my choices--whys, why nots, deep explanations, etc. Essentially, they want me to show all of my steps.

I'm not even halfway done, and this "assignment" has taken several hours and is multiple pages.

Having not experienced this before, I'm concerned that this ask from the company is more so them fishing for information they can feed into an AI. Are assignments of this scale common within the EdTech and Language Tech arena?

Hello all!

I graduated earlier this year with a MS in Human Language Technology from UA. I mention the school because the program is heavily focused on giving students practical, job-like experience. I completed an internship during which I worked on a live project and had the opportunity to present at the Language Technology for All conference in France last year. In addition, I have professional experience in language technology, primarily in text chatbot work that was more manual (e.g. working with patterns that are manually programmed and not inferred using ML) as well as experience with AI and Data Analysis. I have had an impossible time finding a job that is actually relevant and I would really like some advice.

Currently, I am on contract at Meta as a DLA, though I have been tasked with more complex linguistics work, specifically reporting on results of assessments and creating assessment guidelines for various models, both text and voice.

Is there a place I am not looking? I will share my docs in the comments.

Hey everyone, hope u r doing well!

So I obtained an MA in English literature from Hungary (Stipendium Hungaricum)

I have always been passionate by research and would like to pursue a PhD but I realized that literature isn't what I want. I want something more practical, o I am studying another MA in linguistics in my home country.

For my PhD I'd like to transition to something a bit more technical, preferably as technical as possible. I heard about computational linguistics and NLP as the best options for people in my situation. Do you guys have advice please? Where should I start?

Hi there!
I'm a 2nd year student in Applied (Computational) Linguistics. And this year I am supposed to write my first research aka coursework. And it kinda shapes what I will primarily be doing for the next couple of years in uni. At least, it is what I was told there.
In fact, I have no idea what to dive into. Maybe you could recommend some resources or tell about similar experiences of yours. How to choose the topic etc.
I can't understand whether I'm indecisive, don't know enough yet or actually chose not really my thing to study. Hope I'm not alone with such "crisis", though I believe it's not the worst problem that could appear.

Hello there,

I’m a 20-year-old university student in the UK now in his 2nd year of study of a linguistics and literature degree. I am taking an introductory data skills class next semester that is mainly intended to help humanities background students overcome programming anxiety and learn how to utilise linguistics in more STEM-oriented fields, such as corpus linguistic coding, data analysis, etc.

Problem is… Mathematics isn't my strong suit. I dunno why, I’ve kinda just always been more inclined to the Humanities. I passed foundational maths over here, but that stuff is really easy and it's the tier I had to take since I wasn’t good enough for Higher Tier.

But I do wanna learn more! This is a field I'm interested in and that I wanna learn more about. Has anyone from a largely humanities background like me taken a class like this and ever self-taught programming languages or some more complex mathematical skills?

Answers from anyone welcome, but people from the UK who are knowledgeable about our specific academic system would be great!

Thanks in advance! ❤️

Hi, I have a bachelor’s degree in English Language Teaching (ELT), and I’m interested in linguistics. During my university years, I particularly enjoyed my linguistics courses and passed them with good grades. However, my IT skills, like coding, are currently at beginner level. That being said, I believe I can learn to code if I put in the effort.

Since LLMs have become an important part of our daily lives, I believe that computational linguistics and related fields will become even more significant in the future.

Given my academic background, do you think pursuing a career in this field would be a good idea? If so, would a master’s degree in computational linguistics be a good way to get started?

I created a dataset of 1000 authentic SiSwati-English parallel sentences.

Details:

\- Language: SiSwati (ss) / Eswatini Swati, Manzini dialect

\- Speaker: Native (me)

\- Format: CSV with id, english, siSwati, category

\- Categories: family, home, market, school, greetings, love, daily life

\- Quality: Human written, not Google Translated

\- Use case: LLM training, low-resource language research

Why SiSwati? It's a very low-resource language. Good luck finding quality data!

I am selling:

\- Full 1000: $50 (Wise/crypto)

\- Sample 100: $10

Can provide sample of 20 rows before purchase.

DM for sample or to buy. Thanks!

I'm a solo founder building an Arabic veterinary diagnosis assistant for livestock and pets in Egypt (poultry, cattle, buffalo, horses, cats, dogs).

Setup: ~216 English vet textbooks -> 530k chunks (Postgres + pgvector, paraphrase-multilingual-mpnet 768-d with a 128-token window, plus lexical search); Claude Haiku as the LLM.

What we measured:

- Retrieval-first (embed the Arabic symptoms -> top-10 chunks -> LLM) is weak.

- Given the correct passage, the LLM gets ~92% top-1, so the reasoning works.

- "LLM proposes 10 candidate diseases -> search the books by disease name -> LLM picks top-3" reaches 92% top-3 on clean Arabic (MSA) case descriptions.

- On real owner messages from Arabic Q&A sites, agreement is only ~40% exact and ~72% exact or related (labels are the replier's answers, not vet-reviewed).

The translation/dialect problem: owners write in Egyptian colloquial Arabic ("الفراخ بتنهج وفاتحة بوقها", "الجاموسة مانعة الأكل"), while all the evidence is English and organised by disease name. What we tried:

1. AGROVOC Arabic-English glossary: +1 case, and generic terms ("heat", "symptoms") diluted queries.

2. A 287-term Egyptian field-term dictionary: fixed 3 cases, broke 3.

3. A 410-term controlled clinical vocabulary for poultry (MSA + Egyptian): did not meet our gate.

4. LLM rewriting of the whole question into English clinical signs before diagnosis: 38.3% vs 40.4% without, so no gain.

5. BGE-M3 instead of mpnet: not better on our pilot, and 7x slower.

Planned next: self-consistency over candidate lists, 2-3 follow-up questions to separate the top-3, a canonical disease list with hierarchical scoring, and a small paid vet audit.

Questions:

1. What has worked for you mapping dialectal or lay symptom text to clinical concepts? Is the right move to normalise the dialect, or to ask follow-up questions?

2. Do you know Egyptian/Arabic medical NLP resources (dialect symptom lexicons, annotated corpora) or veterinary ontologies (VeNom, SNOMED vet) that can be used?

3. Any pitfalls with LLM-simulated owners for evaluation?

Thanks!

I keep seeing the same pattern in teams working on routing and classification tasks, such as sorting support tickets, tagging documents or assigning intents. The first version is a prompt to a large model with a JSON schema, and it works well enough in the demo. A few months later the complaints are about latency, cost, and the occasional malformed output that breaks something downstream.

Meanwhile, bidirectional encoders keep getting better. ModernBERT handles long context, and few shot methods like SetFit can separate fairly similar labels from a handful of examples per class. On CPU, inference is a fraction of the time of a generative model, and there is no output format to get wrong.

A few things I’d like to hear from people who have actually shipped this:

• Where did an encoder clearly beat prompting an LLM for you, and where did it not?
• How do you handle categories that overlap heavily? Is few shot fine tuning enough, or do you still need a few hundred labelled examples?
• How do you decide when the model should not make the call at all? Do you use confidence thresholds, calibration or something else?

Interested in both the cases where this worked and the ones where you went back to an LLM.

Hi everyone! I’m a Data Science student working on my capstone project on gender-based violence and femicide (GBVF). I’m looking for a dataset of public Twitter/X posts related to GBVF to analyze discussions, identify patterns, and build predictive models. If anyone has an existing dataset or knows where I can access one, I’d really appreciate your help. Thank you!

I've been working on readability formulas, lexical diversity metrics and stylometry for Russian and Spanish texts for a while, and lately I keep asking myself whether this whole area still makes sense

Today you can paste a text into an LLM and ask "how readable is this?" or "who's the likely author?" and get a fluent, often reasonable answer. Next to that, a formula like Flesch reading ease (sentence length plus syllables per word) looks almost naive

Still, I see a few places where plain statistics seem hard to replace:

• Reproducibility: the same text always gets the same number, and you can explain exactly where it came from. An LLM's rating can change between runs, prompts or model versions
• Scale and cost: computing a few hundred features over a million documents takes minutes on a laptop
• Evaluating LLMs themselves: measuring how repetitive, uniform or hard to read generated text is, without asking another model to judge it
• Data pipelines: a lot of pretraining data filtering still relies on simple heuristics like word length, symbol ratios and repetition.
• Research: in stylometry and corpus linguistics, interpretable features are often the whole point

On the other hand, most formulas were calibrated decades ago on small samples of English, and every other language needs its own adaptation, often with questionable results

So I'm curious:

• Do you still use readability, diversity or stylometric metrics in your work? For what?
• Have you replaced them with LLM-based judgments, or do you combine the two?
• Which metrics turned out to be useless in practice, and which ones would you miss?

Hello everyone, I'm a graduate with a Bachelor in English Language Studies (Business English & Corporate Communication track) and I'm looking to apply for the Language Science & Technology (LST) Master's program at Saarland University (Germany) for the next Winter semester.

I don't have a background in technology. My work experience mostly includes English Teaching for a year, then 1.5 year working as an English Communication Specialist for a private university (basically creating website & social contents, managing and translating huge amounts of website articles to English via CMS, working on admission and promotion campaigns & events,...)

But I'm looking to pivot into Tech recently, and after doing research, I'm really interested in Language Technology and found out about this Master program in Germany that is open to admit students with a background in linguistics (which is pretty rare since most program requires you to have a bachelor in computer science).

I know that I need to work extra, extra hard given my background. I've been taking up course on Coursera (Python for Everybody, Linear Algebra, Statistics,...) as the course's advisor advice to increase my chance of admission as well as to feel less overwhelming if I get admitted.

I'll continue to try and learn harder to make up for my lack of knowledge in the field.
Do you guys have any advices for someone looking to get into the field, and also during my time studying the program if I get admitted? Specifically to make learning smoother and to get a better chance at getting hired after graduating would be much appreciated.

Thank you everyone!

I'm a nontechnical founder building a study platform for Brazil's competitive government hiring exams, where candidates take written tests and compete for public sector jobs.

I want to use AI to turn a large bank of past exam questions into short lessons, each focused on one specific topic or skill. My goal is to produce these lessons at high volume while keeping them accurate.

I'm struggling to get AI to identify which questions belong together. Two questions can cover the same subtopic but need different explanations. Calculating a percentage increase and finding the original value after an increase are one example.

Each lesson should have:

• Two concept questions.
• Two new practice questions.
• Two original exam questions, kept unchanged.

The first four questions should prepare the student to solve the final two independently.

With help from AI coding tools, I've built a workflow using existing models, reusable instructions ("skills"), and code functions for extracting documents, pairing questions, and checking results. I run planning, generation, and editorial review in separate AI sessions.

I've tried grouping by subject, text similarity, embeddings, and AI-written skill descriptions. I still get pairs that require different explanations. Stricter matching also removes useful pairs. I've also found that exam questions and official answer keys often don't provide enough information to generate accurate explanations.

In one local batch, six lessons reached editorial review. Only one passed.

Would you start with a detailed topic hierarchy and classify the questions, or identify the knowledge each question requires and build lessons around that?

If you've solved something similar, what worked? A concrete example or tool recommendation would help me understand what to try next.

I'm making a benchmark for ASR systems in my language and I need to start small and expand later. What would you say are priority situations or domains that people expect ASR to work well in?

Like, a meeting in a noisy environment, or a dictation by a single person in a quiet environment. I can then produce transcribed speech recordings that simulate these situations and use them as a test set.

What must an ASR tool be able to work with to be considered a good tool by most people?

Why do embedding models so often use 1536 dimensions specifically?
I understand why hardware-friendly multiples like 64/128/256/512 are desirable. What I’m curious about is the specific choice of 1536 = 3×512.
OpenAI has used 1536-dimensional embeddings, and other vendors also offer/recommend 1536. Is this usually an empirically chosen Goldilocks point between 1024 and 2048—representation quality versus memory/compute—or is there some architectural/hardware reason that makes 1536 particularly convenient?
I’m especially interested in answers from anyone who has actually trained or designed embedding models. I’m not asking why embedding dimensions are generally hardware-aligned; I’m asking why 1536 rather than 1024 or 2048.

Hey guys, im working on a bit of project, and im trying to solve for semantic understanding right now, I am currently deciding between using spaCy for my NER extraction vs something like a BERT.

The general context behind where this is going to be used is intent classification in chat systems, where a text will come in and this layer has to parse out the People in the sentence, the Objects mentioned in the sentence, and the verbs, along with things like quantities and relations between them.

An example of what i mean is,

"The shoes you have delivered to me are red, i asked for black!"

and we then pick out the

1. People involved in this interaction (we cant figure that out from the sentence alone for that we will refer to the handle from which the message was sent)
2. Objects involved - {shoes}
3. Verbs - {delivered}
4. Relationships - {expected colour = black, received colour=red}

Im new to this stuff so maybe im not even asking the right questions, but im hoping that i have done a good enough job of explaining what im doing so that more experienced souls such as yourself may help me.

Thanks 😁

17% of requests tied to one multi-column warranty table were returning 24 months instead of 36, while standard warranty questions kept passing. We traced the failures in Braintrust and saw which retrieved chunks were present when the wrong answer appeared. The chunk boundary had separated the row values from the table heading, so the retrieval context lost the qualifier for 36 months and the reranker favored nearby prose containing 24 months instead. Support had both warranty numbers in separate macros (before the retrieval path was clear), which made the conflicting answers harder to untangle.

Changing the chunking moved those cases in the experiment diff and groundedness improved once the heading stayed with the row. Recall at k barely changed because the table was already being retrieved. We've added the failures to a regression dataset but I'm still concerned about other tables where retrieval looks healthy while structure changes the answer.

What's your approach to catching chunk boundary failures when the right document is already in the candidate set?

Looking for multi-speaker audio where speakers switch between English and Arabic (any dialect, Gulf preferred) within the same conversation, with reference transcripts, ideally with speaker labels and timestamps. It's for evaluating ASR and meeting-transcription quality. Already aware of ESCWA.CS, Mixat and ArzEn. Any others, including licensed or paid ones?
If not of Arabic, any other language combination is fine.

What do you think about the move?