July 2, 2025

news   Read News
Google, OpenAI, Anthropic Crawl Data; Microsoft AI Jobs

Cloudflare is rolling out new tools to help website owners manage how AI companies access and use their content. By default, new websites on Cloudflare will block AI crawlers, requiring AI companies to obtain permission before scraping data. This move aims to protect content creators, as Cloudflare CEO Matthew Prince notes that AI crawlers often scrape far more pages than they generate referrals. For example, Google crawls websites 14-18 times for every referral, while OpenAI's ratio is around 1,500-1,700 to 1, and Anthropic's is even higher at 73,000 to 1. Cloudflare is also testing a 'Pay Per Crawl' system, allowing websites to charge AI companies for access to their content, with major publishers like Condé Nast and Associated Press expressing support. Over 1 million customers have already used Cloudflare's AI-bot-blocking tools since July 2024. In other AI developments, companies like the Bank of New York Mellon are integrating AI employees into their workforce, even giving them email accounts and access to Microsoft Teams. Meanwhile, experts are debating whether the UK workforce has sufficient AI skills, and colleges are adding AI training to courses across various majors to meet employer demand. AI is also impacting job markets, with major tech companies like Microsoft, Google, and Amazon cutting over 77,000 positions in 2025 due to AI automation. However, the World Economic Forum predicts AI will create more jobs than it displaces, emphasizing the need for adaptability and continuous learning. Generative AI is also revolutionizing product development in Europe, enhancing software development productivity and streamlining workflows. In the defense sector, Planet Labs PBC has secured multiple contracts for its AI-enabled solutions, supporting enhanced awareness and decision-making for government customers. Furthermore, educators are finding creative ways to use AI to engage students, such as using AI-powered chatbots to research historical figures.

Key Takeaways

  • Cloudflare now blocks AI crawlers by default for new websites, giving content creators more control over their data.
  • Cloudflare is testing a 'Pay Per Crawl' system, allowing websites to charge AI companies for scraping content.
  • Google's crawl-to-referral ratio is significantly lower (14-18:1) compared to OpenAI (1,500-1,700:1) and Anthropic (73,000:1).
  • Over 1 million Cloudflare customers have used AI-bot-blocking tools since July 2024.
  • Bank of New York Mellon is integrating AI employees with email accounts and Microsoft Teams access.
  • Experts are debating AI skill readiness in the UK workforce, while colleges are adding AI training to various majors.
  • Major tech companies have cut over 77,000 positions in 2025 due to AI automation.
  • The World Economic Forum predicts AI will create 97 million new jobs while displacing 85 million by 2025.
  • Generative AI is boosting software development productivity in Europe.
  • Planet Labs PBC has secured contracts for AI solutions in the Defense & Intelligence sector.

AI secretly grabs web data How to protect your content

AI companies are secretly collecting data from websites to train their AI models. This hurts content creators because people are using AI summaries instead of visiting the original sites. Cloudflare CEO Matthew Prince says that AI sites scrape way more pages than they send visitors to content sites. You can use robots.txt to block some bots, but it's not perfect. Other ways to protect your site include limiting requests and using security measures.

Cloudflare now blocks AI scrapers by default to protect content

Cloudflare is now letting websites automatically block AI companies from taking their data. This new setting requires website owners to give permission before AI bots can collect content. Cloudflare CEO Matthew Prince wants to protect original content online. He believes that if AI companies use data without permission or payment, people won't want to create new content. Cloudflare's network handles about 20 percent of internet traffic and has seen a big increase in AI data crawlers.

Cloudflare blocks AI bots from scraping content without permission

Cloudflare will now ask new website owners if they want to block AI crawlers. This could stop AI chatbots from gathering data to train their models. About 16% of the world's internet traffic goes through Cloudflare. CEO Matthew Prince says this move gives creators more control and helps AI companies innovate. AI crawlers collect data to answer questions, which can take away traffic and money from publishers.

Cloudflare now blocks AI crawlers by default offers pay option

Cloudflare will now block AI web crawlers by default to stop them from using content without permission. New website owners will be asked if they want to allow AI scrapers. Cloudflare is also testing a 'Pay Per Crawl' system where publishers can charge AI scrapers for access. CEO Matthew Prince says people trust AI more, so they aren't reading original content. Cloudflare is working with AI companies to verify their crawlers and state what they're using the content for.

Cloudflare gives tools to control AI training content use

Cloudflare is giving website owners new tools to control if AI bots can use their content for training. Customers can have Cloudflare manage a robots.txt file to block AI crawlers. They can also block AI bots only on parts of their site that make money from ads. Google crawls websites about 14 times for every referral, but OpenAI's crawl-to-referral ratio is 1,700 to 1, and Anthropic's is 73,000 to 1. Since July 2024, over 1 million customers have used Cloudflare's tool to block AI bots.

Cloudflare lets websites charge AI bots for scraping content

Cloudflare is launching a marketplace called Pay per Crawl where websites can charge AI companies to scrape their content. Website owners can choose to let AI crawlers scrape their site for a fee, for free, or block them. New websites on Cloudflare will now block all AI crawlers by default. Cloudflare says that Google's crawler scrapes websites 14 times for every referral, while OpenAI's scrapes 17,000 times and Anthropic's scrapes 73,000 times for every referral.

Cloudflare tool helps websites monetize AI bot crawler access

Cloudflare has launched a tool that lets websites charge AI firms for accessing their content. Website owners can choose if AI crawlers can access their material and set a price for access. This helps them control how their work is used and get paid for it. Major publishers like Condé Nast and Associated Press support the tool. Cloudflare says that Google's ratio of crawls to visitors is 18 to 1, while OpenAI's is 1,500 to 1.

Cloudflare now blocks AI scrapers by default to protect content

Cloudflare is now letting websites automatically block AI companies from taking their data. This new setting requires website owners to give permission before AI bots can collect content. Cloudflare CEO Matthew Prince wants to protect original content online. He believes that if AI companies use data without permission or payment, people won't want to create new content. Cloudflare's network handles about 20 percent of internet traffic and has seen a big increase in AI data crawlers.

Cloudflare blocks AI crawlers by default offers pay option

Cloudflare is now blocking AI web crawlers by default to prevent them from using content without permission. The company is also moving forward with a Pay Per Crawl program that lets customers charge AI companies to scrape their websites. Cloudflare says it can identify even 'shadow' scrapers that are not publicized by AI companies. Over 1 million customer websites have activated Cloudflare's older AI-bot-blocking tools. The company uses behavioral analysis, fingerprinting, and machine learning to classify AI bots.

Millions of websites get new AI bot blocker from Cloudflare

Cloudflare is rolling out a new system that lets millions of websites block AI bots from accessing their content without permission. Eventually, sites will be able to ask for payment from AI firms to have their content scraped. Cloudflare hosts around a fifth of the internet. Roger Lynch, CEO of Condé Nast, said the move was 'a game-changer' for publishers. The company is developing a 'Pay Per Crawl' system to allow content creators to request payment from AI companies.

Cloudflare targets media firms, content creators amid AI boom

Cloudflare is helping media publishers and content creators manage how AI companies access their websites. The services help customers block unauthorized content usage. Cloudflare will now block AI crawlers by default for new customers. Companies can give AI companies access if they choose. Cloudflare is developing a 'Pay Per Crawl' system to request payment from AI companies. The company also rolled out 'AI Labyrinth,' which aims to trap unauthorized AI crawlers.

Cloudflare lets content creators charge fees to AI chatbots

Cloudflare is adding a new feature that lets users charge fees to AI chatbot companies to crawl their sites. Users can allow AI bots full, free access, block all AI chatbots, or charge a fee. AI training crawl activity has increased by 65% over the past 6 months. Cloudflare estimates that about 16% of all internet traffic around the world operates through its network.

Culture is key in the AI age says Cognizant CEO

As AI becomes more common, a company's culture will be its competitive advantage. Companies can capture everyday work into a 'work graph' to let AI learn team rhythms. Context-smart, multi-agent systems keep the cultural DNA alive. AI should enhance a company's unique way of operating. By weaving operational intelligence into AI, businesses can increase productivity and optimize human-machine collaboration.

UK workforce lacks AI expertise experts weigh in

Experts in the UK are discussing whether the general workforce has the skills to use AI properly. They question if people have the technical knowledge or mindset needed. Some say the gap is overstated because there are user-friendly AI tools. Others warn that the UK could fall behind if there isn't targeted training. Experts say leaders need to talk more about how humans can manage and work with AI tools.

AI in media and education is here to stay

Artificial intelligence is becoming more common in media and education. Elizabeth Steinberg shared how she used AI to save time and resources for creating pitches. Students in a UWX AAS degree program created fake content using AI tools to learn about media literacy. Both experiences led to thought-provoking skepticism and critical analysis with an ethical component.

AI is impacting jobs in 2025 what's really happening

AI is eliminating jobs, but promised job creation and retraining programs are mostly theoretical. Major technology companies have cut over 77,000 positions in 2025 because AI is automating roles. Some examples include Microsoft, Google, and Amazon. The World Economic Forum predicts AI will eliminate 92 million jobs while creating 170 million new ones. Some companies are offering retraining, but there's little evidence that society is prepared for this change.

Career strategies to thrive in the AI age

Artificial intelligence is changing every industry, and professionals need to adapt. The World Economic Forum says AI will create 97 million new jobs worldwide by 2025 but displace 85 million roles. To thrive, people need to be adaptable, design their careers like a startup, and integrate humanity into innovation. Visionaries should focus on solving new problems, learning new skills, and leading with insight and creativity.

Bank hires full-time AI employees with email accounts

The Bank of New York Mellon (BNY) has digital employees working with human staff. These AI workers will soon have their own email accounts and participate in Microsoft Teams meetings. BNY has two AI personas one for coding and one for validating payment instructions. These AI workers can access the same tools as human workers and can write and implement patches. BNY aims to give its digital workforce access to email and Microsoft Teams so they can contact human managers when they face a problem.

Colleges add AI skills to courses for all majors

Employers want AI skills for jobs outside of the tech sector. Colleges are adding AI training to their courses to meet this demand. Miami Dade College launched an AI program with classes in machine learning and ethics. Students are looking for ways to boost their AI skills to make themselves more marketable. Colleges are also motivated by these trends and are adding AI to their course catalogs.

Generative AI revolutionizes product development in Europe

Generative AI is changing product development in Europe by boosting software development productivity. It helps teams work faster and with fewer misunderstandings. Generative AI can turn ideas into detailed user stories and scan market trends. It can also generate wireframes and convert instructions into working code. CIOs and CTOs need to rethink culture, governance, and training to integrate AI effectively.

Planet wins contracts for AI solutions in defense sector

Planet Labs PBC has secured four major awards for its new AI-enabled solutions in the Defense & Intelligence sector. These contracts support enhanced awareness and informed decision-making for government customers. Planet won a contract funded by Germany and expanded its contract with the U.S. Department of Defense. Planet also signed an expansion with the U.S. Navy and was selected by NATO for a contract to deliver surveillance and awareness functions.

AI makes history class more fun for students

Heather Brown, an elementary school teacher, uses AI tools to make history class more fun. She uses Magic School AI to create custom chatbots that help students research historical figures. Students can interact with the chatbot and find avenues for further research. The platform can also write lyrics based on the character and their accomplishments. Students can use Genially to create interactive timelines to present their research.

Sources

AI Cloudflare Data Scraping Web Crawlers Content Protection Pay Per Crawl AI Training Robots.txt Matthew Prince Content Creators Website Owners AI Bots Permissions Monetization AI Companies AI Models Internet Traffic AI Innovation Digital Workforce AI Skills Job Market Generative AI Product Development Defense Education AI Ethics Machine Learning AI Tools AI Adoption AI Culture AI in Media AI in Education AI Job Displacement AI Job Creation AI Retraining AI Employees AI Integration AI Solutions AI Chatbots