OilPrice.com

Title: Crude Oil Prices Today | OilPrice.com

Description: Crude oil prices & gas price charts. Oil price charts for Brent Crude, WTI & oil futures. Energy news covering oil, petroleum, natural gas and investment advice

Keywords: oil, price, oil prices, crude oil prices, energy, petroleum, oil gas, oil price, crude oil, oil company, crude oil price, energy news, peak oil, oil price chart, energy price, oil price forecast, natural gas, commodities, BP, oil futures, oil spill, gas prices, Brent crude, WTI, oil and gas investments, oil companies

# robots.txt

#User-agent: Mediapartners-Google
#Disallow:

#
User-agent: *
Disallow: /cgi-bin/
Disallow: /search
Disallow: /goto/
Disallow: /commodity-price-charts?1*
Disallow: /_private/
Disallow: /_vti_bin/
Disallow: /_vti_cnf/
Disallow: /_vti_log/
Disallow: /_vti_pvt/
Disallow: /_vti_txt/
Disallow: /newsletters/
Disallow: /feed/3
Disallow: /newsfeeds/redirect/
Disallow: /salestool
Disallow: /salestool/
Disallow: /articles/sponsored
Disallow: /articles/sponsored/
Disallow: /auth/*
Disallow: /account/*
Sitemap: https://oilprice.com/sitemap_index.xml
Sitemap: https://oilprice.com/sitemap.xml
Sitemap: https://oilprice.com/googlenews.xml

# Original rules for old site
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /component/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/

Disallow: /ajax/csrf
Disallow: /ajax/add_view/*
Disallow: /ajax/return_country
Disallow: /widgets/*
Disallow: /freewidgets/*
Disallow: /opc/*
Disallow: /ajax/get_banner/*
Disallow: /ajax/get_user_blends
Disallow: /ajax/add_pushcrew
Disallow: /ajax/*

#mobile usability problems
#Urgent SEO issues - please confirm receipt 15/02/2017
Disallow: /ajax/get_chart_prices/
Disallow: /commodity-widgets.php
Disallow: /iframes/barcharts
Disallow: /market-intelligence/signup_success/6
Disallow: /newwidgets.php?3
Disallow: /specialreports/8_Mega_Trends.html
Disallow: /widgets/energyproduction.html

Disallow: /*data.json$
Disallow: /*data.json?t=*

Disallow: /*last.json$
Disallow: /*last.json?t=*

Disallow: /Energy/Energy-General/The-Gas-Find-That-Could-Transform-Europes-Energy-Future.html
Disallow: /Energy/Energy-General/This-Could-Be-A-Gamechanger-For-Natural-Gas-In-Europe.html
Disallow: /Energy/Energy-General/How-To-Profit-From-Europes-800-Billion-Energy-Crisis.html
Disallow: /Energy/Energy-General/The-No1-Energy-Stock-for-2024.html
Disallow: /Energy/Energy-General/Forgotten-Gas-Reserves-Could-Be-A-Gamechanger-For-European-Energy.html
Disallow: /Energy/Energy-General/Meet-the-Man-Using-AI-To-Revive-An-Oil-Gas-Play-Supermajors-Left-For-Dead.html
Disallow: /Energy/Natural-Gas/How-Artificial-Intelligence-Could-Trigger-a-Natural-Gas-Boom-in-Europe.html



User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: cohere-ai
User-agent: Diffbot
User-agent: FacebookBot
User-agent: FriendlyCrawler
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: Meta-ExternalAgent
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: omgilibot
User-agent: PerplexityBot
User-agent: Scrapy
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: YouBot
Disallow: /