ESG Data and AI: Sourcing and Structuring for Investment Analysis

ESG data are metrics and information on a company's environmental, social and governance (Environmental, Social, Governance) aspects. For investment analysis, they must be sourced from many sources and brought into a uniform format. AI helps to read out unstructured texts such as sustainability reports, to normalize the information and to assign tags, so that analysts can work with comparable datasets. The evaluation of this data and every investment decision remain with the institution or advisor.

ESG data is difficult to handle for investment analysis because it comes in different formats: as running text in annual reports, as tables in databases, as reports in news or social media. AI-based methods, above all from the field of language processing (Natural Language Processing, or NLP), start exactly here. This article explains how sourcing and structuring work and which limits must be observed.

Why ESG data is difficult to source

A large part of relevant ESG information is contained in unstructured documents. Sustainability reports and annual reports are often very long: the raw data of such XHTML documents sometimes spans several million characters Source. For language models this is a problem, because their input window only processes a limited amount of text (tokens) at once. Therefore it must be filtered out in advance which passages are relevant at all.

In addition, there is no uniform standard against which ESG information could be checked unambiguously. Regulatory frameworks often only describe in general terms what is expected of companies. Professionally defined criteria can therefore vary between evaluators and contain distortions (bias). Algorithms trained on such a basis inherit these limits Source. This is a central reason why AI here takes over the preparation but not the judgment.

How AI sources and structures ESG data

Machine learning can automatically collect ESG data from various sources: from internal documents, public disclosures, third-party databases and unstructured sources such as news or social media. The systems normalize the information and assign tags, so that consistent datasets emerge. This makes it possible to identify relevant data points and metrics without reviewing every document manually Source.

The step from unstructured text to structured data is the core. NLP-based extraction methods convert sentences from sustainability reports into standardized metrics. One project describes how this increases the efficiency and accuracy of data collection and how extensive datasets can thereby be made available for investment analysis Source. Another example: a platform, according to its own statements, analyzes over four million data sources daily, recognizes thousands of public and private companies including their brands and identifiers, and turns unstructured text into structured, further-processable data Source.

Technically, such systems rely on building blocks that are described in more detail in our glossary: documents are broken down into sections (see Chunking), so that matching passages can be found specifically. In extraction, structured output formats help, for example as JSON, so that the results are machine-processable (see Structured Output).

Alternative data sources and their role

Besides reports and ratings, alternative data sources are gaining importance. Rating agencies increasingly integrate such datasets and use machine learning to provide more flexible and up-to-date information. One provider uses NLP to analyze various sources and derive ESG metrics for more than 20,000 companies worldwide on the basis of 150 ESG metrics Source. Another example combines climate science and machine learning to measure and monitor climate risks for market participants Source.

For practice this means: the broader the source base, the more important the clean consolidation becomes. Only when information from reports, databases and news is mapped onto common metrics can companies be meaningfully placed side by side. This comparability is the actual achievement of structuring.

Limits and the role of the human

With ESG data, AI delivers the preparatory work, not the judgment. Studies emphasize that NLP plays a supporting role in finding critical passages in sustainability reports, but that the final review requires humans Source. Missing standards, possible distortions and the risk that language models summarize information incorrectly make control necessary.

For scoreprise.AI this is the clear separation in the area of Investment Intelligence: the AI compiles ESG data and weights it according to the institution's specifications, the advisor makes the decision. This corresponds to the same principle as with portfolio monitoring with AI: observe and prepare instead of evaluate and recommend. So that the prepared information remains traceable, the reference back to the original source is important, a topic that is also central to avoiding AI hallucinations.

Frequently asked questions

Where does the ESG data that AI processes come from?

ESG data comes from several sources: internal documents, public disclosures such as annual and sustainability reports, third-party databases and unstructured sources such as news or social media. AI collects this information, normalizes it and assigns tags, so that a uniform dataset emerges. The breadth of sources is important because individual reports alone are often incomplete.

Why is a language model not enough to read out entire reports?

Sustainability reports are very long, their raw data sometimes spans several million characters. Language models, however, can only process a limited amount of text at once. Therefore documents are broken down into sections in advance and relevant passages are filtered out before the model processes them in structured form.

Can AI give an investment recommendation on ESG data?

No. AI prepares ESG data, it extracts, normalizes and compiles metrics. The evaluation of this data and every investment decision lie with the institution or advisor. Studies emphasize that the final review of the prepared information requires human expertise.

What risks are there with AI-based ESG data processing?

The main risk lies in missing uniform standards and in possible distortions of the underlying criteria. Algorithms inherit the limits of the data they work with. In addition, language models can summarize information incorrectly. Therefore source references and human control are necessary.