Text and data mining: lawful access and opt-out trap

Text and data mining (TDM) means using software to scan large amounts of text or data to find patterns, trends, or correlations. In EU copyright law, TDM has a specific exception, but it is not a free-for-all. The key contrast is between research organisations and everyone else: for non-commercial research by universities or research institutes, the exception is mandatory and cannot be overridden by contract. For commercial users or other purposes, the exception only applies if you have lawful access to the material and the rights holder has not placed a machine-readable opt-out, such as a robots.txt file or metadata that blocks mining.

To avoid traps, remember three things. First, 'lawful access' means you must have a subscription, licence, or open-access right to the content—you cannot mine pirated copies. Second, the opt-out must be machine-readable; a simple terms-of-service clause is not enough unless it is coded in a way automated tools can detect. Third, for research organisations, the opt-out is invalid, so if a publisher tries to block a university’s mining, the exception overrides it. A quick way to test this: ask yourself whether the user has a legal right to view the content and whether the rights holder has set a technical barrier like a robots.txt file.

To lock this in, picture a researcher mining a paywalled journal database. If the university has a subscription, the researcher has lawful access, and the publisher cannot opt out because it is non-commercial research. Now imagine a startup mining the same database for market trends: the startup needs lawful access and must check for a machine-readable opt-out. If the publisher has a robots.txt file blocking mining, the startup cannot rely on the exception. This contrast makes the rule stick.