Deep Web Search Engines
Where to start a deep web search is easy. You hit Google.com and when you brick wall it, you go to scholar.google.com which is the academic database of Google.
After you brick wall there, your true deep web search begins. You need to know something about your topic in order to choose the next tool. To be fair, some of these sites have improved their index-ability with Google and are now technically no longer Deep Web, rather kind-of-deep-web. However, there are only a few that have done so. I recommend you use your browsers ‘search’ option to locate on this page your topic of interest, as the page has gotten long.
To all the 35F and 35G’s out there at Fort Huachuca and elsewhere, you will find some useful links here to hone in on your AO.
Zuula.com – nice multi engine aggregator
Surfwax – They have a 2011 interface for rss and a 2009 interface I think is better. Takes 60 seconds to understand how to use it.
Dogpile – another multi engine aggregator
Scout Project- scout.wisc.edu — Since 1994, the Scout Project has focused on developing better tools and services for finding, filtering, and presenting online information and metadata.
www.findsmarter.com – You can filter the search by domain extension, or by topic which is quite neat. Sources Yahoo, Bing, Wiki, Blekko, and Alltheweb
Yippy.com – A useful, non-graphical clustering of results. Give it 2 minutes of your time to understand how it works and it will give back hours of saved research time.
www.quintura.com – An interesting data mining search engine that shows related words to your target. Good if you aren’t sure what the exact name of the subject is you seek
www.navagent.com/ - Not a web based search engine, requires you download software. Highly rated, very interesting especially to the 35F intel types.
Archive.org – Huge behemoth of media now public domain – rare books, sound recordings, video, 20 year archived images of all old websites, and free audio books! Makes my top 10 list. (and my top 3)
WWW virtual library – a listing of indexes to industries. Need to know about Architecture? Biochemical war? Zoology? This may get you there.
FindArticles.com – FindArticles has articles from about 500 periodicals with coverage back to 1998, and is completely free of charge.
Library of Congress – loc.gov – Phenomenal digitized archives, “American Memory” especially interesting. Includes a good newspaper archive.
National Security Archive – Declassified papers and such. In their words – “National Security Archive Electronic Briefing Books provide online access to critical declassified records on issues including U.S. national security, foreign policy, diplomatic and military history, intelligence policy, and more. Updated frequently, the Electronic Briefing Books represent just a small sample of the documents in our published and unpublished collections.”
www.osti.gov - Government research archives, if your tax dollars paid for it, the results are here. Also a huge collection of science presentation videos.
US Geologic Survey – Imagery and Maps galore. 3 portals to fetch data, EarthExplore, Glovis and Seamless. Extremely complex. There are tutorials on how to get free aerial photos over at learninggis.com.
US National Map by USGS - The source for current geospatial data from the USGS.
http://adswww.harvard.edu/ – Physics and Astronomy data engine for academic papers
http://databank.worldbank.org/data/home.aspx – Specialty statistical data on all kinds of subjects, from countries GDP to levels of blindness.
http://www.quandl.com/ – An awesome collection of 9,000,000 of financial, economic, and social datasets.
Smithsonian Institution Libraries — 20 libraries from museum complexes around the world.
The National Archives — National Archives’ research tools and online databases.
HighWire Press — Online catalog of the largest repository of free full-text and non-free text, peer-reviewed content, from over 1000 different journals.
Education Resources Information Center (ERIC) — A catalog with more than 1.2 million bibliographic records, many with full text. Sponsored by the US Department of Education.
Encyclopædia Britannica — The old and authoritative encyclopedia searchable with full text online. No longer printing paper copies. The internet killed the library star.
Topix.net — A news search engine.
Internet Public Library — Internet’s public library. More for kids than adults, IMHO.
San Francisco Public Library – A great online library. This is just one example of many such local public libraries that offer similar services. Sorry, you can’t use their access to commercial archives unless you are a member.
Xrefer (commercial) — Fee based database of 236 titles and over 2.9 million entries.
LexisNexis (commercial) — Billed as the world’s largest collection of public records, unpublished opinions, legal, news, and business information. Over 35,000 individual sources are claimed as searchable. I’ve not been able to justify subscribing, so I don’t know.
Forrester Research (Commercial) — An independent technology and market research company, publishing in-depth research reports on a variety of subjects.
Factiva (Commercial) — Online collection of about 10,000 individual sources. It’s a fact.
zabasearch.com – finding people
Intellus (commercial) – Finding people plus background checks on people and other features.
US Search (commercial) – Finding people plus background checks on people.
123 People (commercial) – A multi search engine built around finding people.
Integrascan – Finding people plus background checks on people.
State of Texas DOT Criminal Background Check – The central background check for felonies provided by the state. Most misdemeanours don’t show up.
socialcatfish.com – Has background check capability plus a photo reverse lookup
Hathi Trust – http://www.hathitrust.org/ — a partnership of major research institutions and libraries working to ensure that the cultural record is preserved and accessible long into the future. There are more than sixty partners in HathiTrust, and membership is open to institutions worldwide.
Books.google.com – They are putting the squeeze on all the book scanning businesses. They want to scan the world to add it to the Google Borg. You will be assimilated.
The Online Books Page — A searchable database of more than 28,000 English works with full text available for free online.
Bibliomania — A database of free literature from more than 2,000 classic texts. Archive.org crushes this.
Project Gutenberg — The granddaddy of online books with a catalog of more than 20,000 free books with full text available online. Included in Archive.org.
The National Academies Press — Only about 3,000 free books online and ~900 for-sale PDFs.
ebrary — A database of about 20,000 full-text books. Focusing on academia and business.
UNZ – An odd collection of periodicals and book scanned.
Get Abstracts (commercial) — Large online library of more than 8,000 business book summaries. It is the most efficient way to get the best business titles.
Getty Research Institute – http://www.getty.edu/research/library/ – The Getty Research Institute library collections include over one million books, periodicals, study photographs, and auction catalogs as well as extensive special collections of rare and unique materials. Focusing on art history, architecture, and related fields, they begin with the archaeology of prehistory and extend to the contemporary moment.
Library of Congress Newspaper Resource List – LOC does a great job getting the list together of wonderful newspaper archives.
NewspaperArchive.com – (commercial) – Known for a large collection. I haven’t used it, so I can’t confirm this.
xooxleanswers.com – Great list of newspaper archives from Xooxle. Good list and a funky name. Two thumbs up.
University of Penn Newspaper Archive – List of US newspaper archives and dates. Looked like a deeper list of Texas newspapers, so this effort may be a deep comprehensive list.
Australia Newspaper archives – http://trove.nla.gov.au/ndp/del/about – A phenomenal newspaper archive of most Aussie newspapers from 1820’s going to 1980’s. Including Taz as well.
Agency ComPile — Advertising and marketing agencies.
Alibaba – An international marketplace of businesses looking for businesses.
Kompass — Business to business search engine.
GPO Access Economic Indicators — Gateway for economic indicators from January 1998 to present.
Government Printing Office — Big catalog of stuff published by the Government Printing office. Has business stuff but much much more. Environmental reports, legal docs, nature stuff. Hell, I typed in ‘mushroom’ and pulled up 34 entries.
Hoover’s — The Big Boy of info on businesses.
ThomasNet — Just an industrial product search directory.
SBA Loan Data — Loan program approval activity from the Small Business Administration.
GuideStar.org — A searchable database of non-profit organizations including 501c.
Melissa Data — Comprehensive directory of demographic data, sortable by ZIP code.
Kelley Blue Book — A guide to pricing new and used vehicles.
Edmunds — A recognized and established guide to pricing new and used vehicles.
Consumer Reports — (commercial) A trusted guide to product reviews, including autos, appliances, electronics, computers, personal finance, etc. I use it.
FreeLunch.com —Directory of free economic data.
Bureau of Labor Statistics — Job-based and consumer economic info from the US Department of Labor.
Salary Wizard Calculator — Tools that shows national average salaries adjusted by location for different jobs.
Economagic — A data directory containing over 200,000 econ files.
Penn World Tables — National income data for all countries for the years 1950-2007.
America’s Job Bank — Database of jobs and resumes.
USAJOBS — Portal of data on federal government jobs.
Regional Economic Conditions (RECON) — Economic data available by state, county, and MSA.
InvestIQ — Market data from around the world in regions.
BigCharts — Quotes and performance charts on different stocks and mutual funds.
SmartMoney.com Tools — A portal of for stock analysis tools.
NASDAQ Trader — A database of stock data from the NASDAQ stock exchange.
SEC Info — EDGAR and SEC filings searchable by name, industry, SIC code, etc.
EDGAR Online — SEC filings searchable by ticker or company name.
American FactFinder — Aggregate census bureau data to be searched by city, county, or ZIP code.
FedStats — Gateway for statistics on 100 US federal agencies.
United States Patent and Trademark Office — Database of patents, full-text and full-page images.
Historical Census Browser — Repository of historical US census data. Going back to 1790 compiled by the University of Virginia.
Geospatial One Stop — Awesome GIS data warehouse of geographic data, shapefiles, imagery, and displayable on maps.
Grants.gov — Grant opportunities, from everything under the sun.
Technology Opportunities Program Grants Database — Listing of technology grants, peruse by keyword, state, and year.
United States Government Printing Office (GPO) — I mentioned this earlier, they seem to have everything. A search engine for mutliple government databases: US budgets, campaign reform hearings, code of federal regulations, congressional bills, etc
CIA Electronic Reading Room — The usual uninteresting declassified CIA documents.
POW/MIA Databases and Documents — Info on POWs and MIAs.
ZIP+4 Lookup — US ZIP codes and ZIP+4 codes
FIRST — http://first.sipri.org/ Military aggression database and weapons holdings.
Economics of Tobacco Control — Information regarding tobacco usage and policy for180 countries.
Country Indicators for Foreign Policy —Statistical tables on countries’ foreign policies.
World Bank Data — Key development data and statistics for countries and worldwide groups.
CIA Factbook — Reference materials containing information on every country in the world.
US International Trade Statistics — International trade statistics,by country or type of good.
US Foreign Trade Highlights — Information of US international trade.
Energy Information Administration International Energy Data and Analysis — Energy balances sorted by country. Explains why the US gives huge amounts of cash to oil producing countries.
Law Library of Congress — Allegedly, the largest collection of legal materials in the world, over 2 million volumes.
Global Legal Information Network — Laws, regulations, judicial decisions, and other legal sources.
FindLaw — Free legal database, with collections of cases and codes, legal news.
Office of Postsecondary Education Security Statistics — Contains college campus crime statistics, sortable.
Bureau of Justice Statistics — http://bjs.ojp.usdoj.gov/ Legal and judicial statistics, everything from crime to law enforcement.
The Avalon Project at Yale Law School — Documents in law, history, and diplomacy.
US Congressional Documents and Debates, 1774–1873 (Library of Congress) — The historical Congressional records, bills, documents, statutes, journals, and debates from LOC.
Lobbying Database — Who spent what on firms who have spent lobbying money from 1998. The US, with the finest Congress money can buy!
Legislative Activities — Synopsis of House of Representatives’ bill summary, status, public laws, and votes.
heinonline.org – (commercial) – Claims to be the ‘worlds largest image based database of legal documents’ . I was able to find an obtuse document on using Bayes Theorem for fact finding in a criminal case.
Project Vote Smart — Government officials and election candidates database, order by last name or ZIP code.
National Institute for Health Research Archive — http://www.nihr.ac.uk/Pages/NIHRArchive.aspx Database of ongoing or completed projects funded by the British NHS.
National Institutes of Health — Encyclopedia of health topics. More of a kindof-deep-web resource, as Google has this indexed.
American Hospital Directory — Index of US hospital information.
Globalhealthfacts.org — Searchable world health information, by country, disease, condition, program, or demographic. Quickly lay out the conditions in a country.
Centers for Disease Control and Prevention Data and Statistics — Statistical health information according to the CDC. Not true deep web, but useful.
New England Journal of Medicine (commercial) — A Leading medical journal with full text past issues available online. Commericial, but you can access most for free.
ScienceResearch.com — Searchable access to scientific journals and databases.
Academic Index – Main search is a filtered Google search aimed at high authority rank sites, mainly .edu and .gov which filters a great deal out. Second search ties into deep web academic and non-academic databases skewed to librarians and educators.
Science.gov — Gateway to science info provided by US government agencies.
VideoLectures.net - Phenomenal video lecture coverage from high authority rank sources. A great go-to place to find peer-reviewed, conference presented, in depth coverage of a topic at a conference. A nice bonus, is the presentation slides are shown separately, and you can jump to slides of interest to you. Heavily technology based, and 66% is in English. Most lectures 45 minutes or longer.
WebCASPAR — A horrible interface to an alleged wealth of statistical info on science and engineering. I found the site slow, cludgy and designed around 1965 run off of candle power. From their website:”The WebCASPAR database provides easy access to a large body of statistical data resources for science and engineering (S&E) at U.S. academic institutions. WebCASPAR emphasizes S&E, but its data resources also provide information on non-S&E fields and higher education in general. ”
The Complete Work of Charles Darwin — Charles Darwin’s published works, search-able and available online. He’s still old and his works still ramble. Scanning didn’t help him much.
USGS Real-Time Water Data — Real-time map of streamflow and water quality data of the USA’s rivers and reservoirs.
USGS Earthquake Hazards Program — Showing real-time earthquake data. Focus on US but covers world as well.
IEEE Publications (Commercial) — Contains over 1.4 million documents from the Institute of Electronics and Electrical Engineers.
Society of Petroleum Engineers Archive (Commercial) — Petroleum engineers technical papers.
Arxiv – arxiv.org/ — Cornell University repository. Access to 700,000+ technical papers on everything from quantitative biology to computer science. Appears to offer full text in several formats.
VADLO – www.vadlo.com/ — Life Science Search Engine. Very hit and miss. Don’t have high expectations.
Deep Dyve (Commercial) – www.deepdyve.com DeepDyve has aggregated millions of articles across thousands of journals from the world’s leading publishers, including Springer, Nature Publishing Group, Wiley-Blackwell and more. Haven’t paid the premium to give it a test ride, if someone has, please write a review below.
Data Mining Data sources - http://www.kdnuggets.com/datasets/index.html – Links to gobs of free and commercial datasets used for data mining.
scirus.com — searches some of the other deep web sites. Doesn’t have access to it’s own proprietary repository. Does a good job aggregating sources, however.
NTSB Accident Database and Synopses — The National Transportation Safety Board’s database of aviation accidents, ranging from 1962 to present.
NTSB Aviation Accident Database— Aviation accident data from the National Transportation Safety Board
National Highway Traffic Safety Administration — A database of car and car part defects, searchable by item number or car make and model.
SaferCar.gov — Crash test safety ratings for automobiles since 1990.
flightwise.com — Real-time flight tracking, with support for Google Earth.
FlightAware — Fee flight tracking with history, graphs, and maps.
Security Tube – securitytube.net — A large library of videos covering many topics in InfoSec, cyberwar, and most of the hacking conferences.
DefCon — The main hackers Con, so well known that now the Feds send their folks here and it has become a wild west training ground for coming trends. Archives go back to Defcon 1. They are now on Defcon 20, I think.
semanticommunity.info- General military hardware overview. Some deeper links to deep web
Hey people, we are cool folks here! If you know of a useful deep web resource, put a comment below and share the love!
After you brick wall there, your true deep web search begins. You need to know something about your topic in order to choose the next tool. To be fair, some of these sites have improved their index-ability with Google and are now technically no longer Deep Web, rather kind-of-deep-web. However, there are only a few that have done so. I recommend you use your browsers ‘search’ option to locate on this page your topic of interest, as the page has gotten long.
To all the 35F and 35G’s out there at Fort Huachuca and elsewhere, you will find some useful links here to hone in on your AO.
If you find a bad link, Comment the link below. I try to keep this up to date, but only with your help!
Last updated SEP 23, 2014
Multi Search engines
Deeperweb.com – This is my favorite search engine. It breaks your results down into categories – general web, blogs, news, academic, cloud, metrics, research, etc. This allows you to quickly focus on the type of answer you were looking for. Makes my top 10 websites!Zuula.com – nice multi engine aggregator
Surfwax – They have a 2011 interface for rss and a 2009 interface I think is better. Takes 60 seconds to understand how to use it.
Dogpile – another multi engine aggregator
Scout Project- scout.wisc.edu — Since 1994, the Scout Project has focused on developing better tools and services for finding, filtering, and presenting online information and metadata.
www.findsmarter.com – You can filter the search by domain extension, or by topic which is quite neat. Sources Yahoo, Bing, Wiki, Blekko, and Alltheweb
Cluster Analysis Engine
TouchGraph – A brilliant clustering tool that shows you relationships in your search results using a damn spiffy visualization. The smart way to use it, is to let it help you find new sources to your search topic. I have to add, the wiggly effect on the visualization is damn cool, just grab the center item and move it to understand what I’m talking about. (sometimes it doesn’t wiggle, however. Java issue?)Yippy.com – A useful, non-graphical clustering of results. Give it 2 minutes of your time to understand how it works and it will give back hours of saved research time.
www.quintura.com – An interesting data mining search engine that shows related words to your target. Good if you aren’t sure what the exact name of the subject is you seek
www.navagent.com/ - Not a web based search engine, requires you download software. Highly rated, very interesting especially to the 35F intel types.
Speciality Deep Web Engines
Infomine - comprehensive virtual library and reference tool for academics. But I’ve found it not too useful, better engines out there.Archive.org – Huge behemoth of media now public domain – rare books, sound recordings, video, 20 year archived images of all old websites, and free audio books! Makes my top 10 list. (and my top 3)
WWW virtual library – a listing of indexes to industries. Need to know about Architecture? Biochemical war? Zoology? This may get you there.
FindArticles.com – FindArticles has articles from about 500 periodicals with coverage back to 1998, and is completely free of charge.
Library of Congress – loc.gov – Phenomenal digitized archives, “American Memory” especially interesting. Includes a good newspaper archive.
National Security Archive – Declassified papers and such. In their words – “National Security Archive Electronic Briefing Books provide online access to critical declassified records on issues including U.S. national security, foreign policy, diplomatic and military history, intelligence policy, and more. Updated frequently, the Electronic Briefing Books represent just a small sample of the documents in our published and unpublished collections.”
www.osti.gov - Government research archives, if your tax dollars paid for it, the results are here. Also a huge collection of science presentation videos.
US Geologic Survey – Imagery and Maps galore. 3 portals to fetch data, EarthExplore, Glovis and Seamless. Extremely complex. There are tutorials on how to get free aerial photos over at learninggis.com.
US National Map by USGS - The source for current geospatial data from the USGS.
http://adswww.harvard.edu/ – Physics and Astronomy data engine for academic papers
http://databank.worldbank.org/data/home.aspx – Specialty statistical data on all kinds of subjects, from countries GDP to levels of blindness.
http://www.quandl.com/ – An awesome collection of 9,000,000 of financial, economic, and social datasets.
General
GPO’s Catalog of US Government Publications — Federal publications database.Smithsonian Institution Libraries — 20 libraries from museum complexes around the world.
The National Archives — National Archives’ research tools and online databases.
HighWire Press — Online catalog of the largest repository of free full-text and non-free text, peer-reviewed content, from over 1000 different journals.
Education Resources Information Center (ERIC) — A catalog with more than 1.2 million bibliographic records, many with full text. Sponsored by the US Department of Education.
Encyclopædia Britannica — The old and authoritative encyclopedia searchable with full text online. No longer printing paper copies. The internet killed the library star.
Topix.net — A news search engine.
Internet Public Library — Internet’s public library. More for kids than adults, IMHO.
San Francisco Public Library – A great online library. This is just one example of many such local public libraries that offer similar services. Sorry, you can’t use their access to commercial archives unless you are a member.
Xrefer (commercial) — Fee based database of 236 titles and over 2.9 million entries.
LexisNexis (commercial) — Billed as the world’s largest collection of public records, unpublished opinions, legal, news, and business information. Over 35,000 individual sources are claimed as searchable. I’ve not been able to justify subscribing, so I don’t know.
Forrester Research (Commercial) — An independent technology and market research company, publishing in-depth research reports on a variety of subjects.
Factiva (Commercial) — Online collection of about 10,000 individual sources. It’s a fact.
File and Metadata Discovery
www.findthatfile.com – locates files even if embedded or hidden in a .zip format. Also checks podcasts, FTP and other venues.Find People & Background Checks
Pipl.com– for finding peoplezabasearch.com – finding people
Intellus (commercial) – Finding people plus background checks on people and other features.
US Search (commercial) – Finding people plus background checks on people.
123 People (commercial) – A multi search engine built around finding people.
Integrascan – Finding people plus background checks on people.
State of Texas DOT Criminal Background Check – The central background check for felonies provided by the state. Most misdemeanours don’t show up.
socialcatfish.com – Has background check capability plus a photo reverse lookup
Books Online
Archive.org – Has books online in epub, txt, and pdf formats. The collection encompasses others such as Gutenberg Press, etc. So this is the best site to start with. Again, this makes my top 10 websites. Share the love.Hathi Trust – http://www.hathitrust.org/ — a partnership of major research institutions and libraries working to ensure that the cultural record is preserved and accessible long into the future. There are more than sixty partners in HathiTrust, and membership is open to institutions worldwide.
Books.google.com – They are putting the squeeze on all the book scanning businesses. They want to scan the world to add it to the Google Borg. You will be assimilated.
The Online Books Page — A searchable database of more than 28,000 English works with full text available for free online.
Bibliomania — A database of free literature from more than 2,000 classic texts. Archive.org crushes this.
Project Gutenberg — The granddaddy of online books with a catalog of more than 20,000 free books with full text available online. Included in Archive.org.
The National Academies Press — Only about 3,000 free books online and ~900 for-sale PDFs.
ebrary — A database of about 20,000 full-text books. Focusing on academia and business.
UNZ – An odd collection of periodicals and book scanned.
Get Abstracts (commercial) — Large online library of more than 8,000 business book summaries. It is the most efficient way to get the best business titles.
Getty Research Institute – http://www.getty.edu/research/library/ – The Getty Research Institute library collections include over one million books, periodicals, study photographs, and auction catalogs as well as extensive special collections of rare and unique materials. Focusing on art history, architecture, and related fields, they begin with the archaeology of prehistory and extend to the contemporary moment.
Newspaper Archives Online
US newspaper coverage – ResearchGuides lists links to many newspapersLibrary of Congress Newspaper Resource List – LOC does a great job getting the list together of wonderful newspaper archives.
NewspaperArchive.com – (commercial) – Known for a large collection. I haven’t used it, so I can’t confirm this.
xooxleanswers.com – Great list of newspaper archives from Xooxle. Good list and a funky name. Two thumbs up.
University of Penn Newspaper Archive – List of US newspaper archives and dates. Looked like a deeper list of Texas newspapers, so this effort may be a deep comprehensive list.
Australia Newspaper archives – http://trove.nla.gov.au/ndp/del/about – A phenomenal newspaper archive of most Aussie newspapers from 1820’s going to 1980’s. Including Taz as well.
Audio Books Online
LibreVox.org – Huge selection of audio recordings and AudioBooks read by volunteer voice artistsVideos
www.liveleak.com – A Video news aggregator of citizen supplied videos. Great for OSInt in foreign countries. The site appears to have some good aggregation functions to turn randomly submitted videos into a logical collection around a topic. Proven useful for aggregating the Feb 15, 2013 meteorite strike in Russia.Business Deep Web Engines
AAAAgencySearch.com — Advertising agencies via the American Association of Advertising Agencies.Agency ComPile — Advertising and marketing agencies.
Alibaba – An international marketplace of businesses looking for businesses.
Kompass — Business to business search engine.
GPO Access Economic Indicators — Gateway for economic indicators from January 1998 to present.
Government Printing Office — Big catalog of stuff published by the Government Printing office. Has business stuff but much much more. Environmental reports, legal docs, nature stuff. Hell, I typed in ‘mushroom’ and pulled up 34 entries.
Hoover’s — The Big Boy of info on businesses.
ThomasNet — Just an industrial product search directory.
SBA Loan Data — Loan program approval activity from the Small Business Administration.
GuideStar.org — A searchable database of non-profit organizations including 501c.
Consumer Engines
US Consumer Products Safety Commission Recalled Products — Listing of products, sortable by company name.Melissa Data — Comprehensive directory of demographic data, sortable by ZIP code.
Kelley Blue Book — A guide to pricing new and used vehicles.
Edmunds — A recognized and established guide to pricing new and used vehicles.
Consumer Reports — (commercial) A trusted guide to product reviews, including autos, appliances, electronics, computers, personal finance, etc. I use it.
Economic and Job Search Engines
EUROPA Press Release Database — Database of press releases by the European Union.FreeLunch.com —Directory of free economic data.
Bureau of Labor Statistics — Job-based and consumer economic info from the US Department of Labor.
Salary Wizard Calculator — Tools that shows national average salaries adjusted by location for different jobs.
Economagic — A data directory containing over 200,000 econ files.
Penn World Tables — National income data for all countries for the years 1950-2007.
America’s Job Bank — Database of jobs and resumes.
USAJOBS — Portal of data on federal government jobs.
Regional Economic Conditions (RECON) — Economic data available by state, county, and MSA.
Finance
Bankrate.com — Database of interest rates for different loan types, mortgages, and savings accounts.InvestIQ — Market data from around the world in regions.
BigCharts — Quotes and performance charts on different stocks and mutual funds.
SmartMoney.com Tools — A portal of for stock analysis tools.
NASDAQ Trader — A database of stock data from the NASDAQ stock exchange.
SEC Info — EDGAR and SEC filings searchable by name, industry, SIC code, etc.
EDGAR Online — SEC filings searchable by ticker or company name.
Government Search Engines
Copyright Records (LOCIS) — Online copyright records, documents, serials, and multimedia.American FactFinder — Aggregate census bureau data to be searched by city, county, or ZIP code.
FedStats — Gateway for statistics on 100 US federal agencies.
United States Patent and Trademark Office — Database of patents, full-text and full-page images.
Historical Census Browser — Repository of historical US census data. Going back to 1790 compiled by the University of Virginia.
Geospatial One Stop — Awesome GIS data warehouse of geographic data, shapefiles, imagery, and displayable on maps.
Grants.gov — Grant opportunities, from everything under the sun.
Technology Opportunities Program Grants Database — Listing of technology grants, peruse by keyword, state, and year.
United States Government Printing Office (GPO) — I mentioned this earlier, they seem to have everything. A search engine for mutliple government databases: US budgets, campaign reform hearings, code of federal regulations, congressional bills, etc
CIA Electronic Reading Room — The usual uninteresting declassified CIA documents.
POW/MIA Databases and Documents — Info on POWs and MIAs.
ZIP+4 Lookup — US ZIP codes and ZIP+4 codes
International Search Engines
International Data Base (IDB) — Statistical tables and demographic information for countries and areas.FIRST — http://first.sipri.org/ Military aggression database and weapons holdings.
Economics of Tobacco Control — Information regarding tobacco usage and policy for180 countries.
Country Indicators for Foreign Policy —Statistical tables on countries’ foreign policies.
World Bank Data — Key development data and statistics for countries and worldwide groups.
CIA Factbook — Reference materials containing information on every country in the world.
US International Trade Statistics — International trade statistics,by country or type of good.
US Foreign Trade Highlights — Information of US international trade.
Energy Information Administration International Energy Data and Analysis — Energy balances sorted by country. Explains why the US gives huge amounts of cash to oil producing countries.
Law and Politics
THOMAS (Library of Congress) — Legislative information from the Library of Congress.Law Library of Congress — Allegedly, the largest collection of legal materials in the world, over 2 million volumes.
Global Legal Information Network — Laws, regulations, judicial decisions, and other legal sources.
FindLaw — Free legal database, with collections of cases and codes, legal news.
Office of Postsecondary Education Security Statistics — Contains college campus crime statistics, sortable.
Bureau of Justice Statistics — http://bjs.ojp.usdoj.gov/ Legal and judicial statistics, everything from crime to law enforcement.
The Avalon Project at Yale Law School — Documents in law, history, and diplomacy.
US Congressional Documents and Debates, 1774–1873 (Library of Congress) — The historical Congressional records, bills, documents, statutes, journals, and debates from LOC.
Lobbying Database — Who spent what on firms who have spent lobbying money from 1998. The US, with the finest Congress money can buy!
Legislative Activities — Synopsis of House of Representatives’ bill summary, status, public laws, and votes.
heinonline.org – (commercial) – Claims to be the ‘worlds largest image based database of legal documents’ . I was able to find an obtuse document on using Bayes Theorem for fact finding in a criminal case.
Project Vote Smart — Government officials and election candidates database, order by last name or ZIP code.
Medical and Health
PubMed — The U.S. National Library of Medicine contains over 16 million citations from MEDLINE and other life science journals going back to the 1950s. Contains links to full-text articles and external resources. Supposed to be the best damn resource for medical out there.National Institute for Health Research Archive — http://www.nihr.ac.uk/Pages/NIHRArchive.aspx Database of ongoing or completed projects funded by the British NHS.
National Institutes of Health — Encyclopedia of health topics. More of a kindof-deep-web resource, as Google has this indexed.
American Hospital Directory — Index of US hospital information.
Globalhealthfacts.org — Searchable world health information, by country, disease, condition, program, or demographic. Quickly lay out the conditions in a country.
Centers for Disease Control and Prevention Data and Statistics — Statistical health information according to the CDC. Not true deep web, but useful.
New England Journal of Medicine (commercial) — A Leading medical journal with full text past issues available online. Commericial, but you can access most for free.
Science and Academic
(Scholarly Paper search engines now have their own page)ScienceResearch.com — Searchable access to scientific journals and databases.
Academic Index – Main search is a filtered Google search aimed at high authority rank sites, mainly .edu and .gov which filters a great deal out. Second search ties into deep web academic and non-academic databases skewed to librarians and educators.
Science.gov — Gateway to science info provided by US government agencies.
VideoLectures.net - Phenomenal video lecture coverage from high authority rank sources. A great go-to place to find peer-reviewed, conference presented, in depth coverage of a topic at a conference. A nice bonus, is the presentation slides are shown separately, and you can jump to slides of interest to you. Heavily technology based, and 66% is in English. Most lectures 45 minutes or longer.
WebCASPAR — A horrible interface to an alleged wealth of statistical info on science and engineering. I found the site slow, cludgy and designed around 1965 run off of candle power. From their website:”The WebCASPAR database provides easy access to a large body of statistical data resources for science and engineering (S&E) at U.S. academic institutions. WebCASPAR emphasizes S&E, but its data resources also provide information on non-S&E fields and higher education in general. ”
The Complete Work of Charles Darwin — Charles Darwin’s published works, search-able and available online. He’s still old and his works still ramble. Scanning didn’t help him much.
USGS Real-Time Water Data — Real-time map of streamflow and water quality data of the USA’s rivers and reservoirs.
USGS Earthquake Hazards Program — Showing real-time earthquake data. Focus on US but covers world as well.
IEEE Publications (Commercial) — Contains over 1.4 million documents from the Institute of Electronics and Electrical Engineers.
Society of Petroleum Engineers Archive (Commercial) — Petroleum engineers technical papers.
Arxiv – arxiv.org/ — Cornell University repository. Access to 700,000+ technical papers on everything from quantitative biology to computer science. Appears to offer full text in several formats.
VADLO – www.vadlo.com/ — Life Science Search Engine. Very hit and miss. Don’t have high expectations.
Deep Dyve (Commercial) – www.deepdyve.com DeepDyve has aggregated millions of articles across thousands of journals from the world’s leading publishers, including Springer, Nature Publishing Group, Wiley-Blackwell and more. Haven’t paid the premium to give it a test ride, if someone has, please write a review below.
Data Mining Data sources - http://www.kdnuggets.com/datasets/index.html – Links to gobs of free and commercial datasets used for data mining.
scirus.com — searches some of the other deep web sites. Doesn’t have access to it’s own proprietary repository. Does a good job aggregating sources, however.
Transportation
FAA Flight Delay Information — A map of the United States with flight delay information from the nation’s largest airports.NTSB Accident Database and Synopses — The National Transportation Safety Board’s database of aviation accidents, ranging from 1962 to present.
NTSB Aviation Accident Database— Aviation accident data from the National Transportation Safety Board
National Highway Traffic Safety Administration — A database of car and car part defects, searchable by item number or car make and model.
SaferCar.gov — Crash test safety ratings for automobiles since 1990.
flightwise.com — Real-time flight tracking, with support for Google Earth.
FlightAware — Fee flight tracking with history, graphs, and maps.
History
Ohio State – Compiles lots of Civil War info on troop movements, camps, battles, etc.Cyber War / InfoSec
Iron Geek irongeek.com — An excellent library of videos explaining many facets of InfoSec and hacking & securitySecurity Tube – securitytube.net — A large library of videos covering many topics in InfoSec, cyberwar, and most of the hacking conferences.
DefCon — The main hackers Con, so well known that now the Feds send their folks here and it has become a wild west training ground for coming trends. Archives go back to Defcon 1. They are now on Defcon 20, I think.
semanticommunity.info- General military hardware overview. Some deeper links to deep web
Hey people, we are cool folks here! If you know of a useful deep web resource, put a comment below and share the love!
Discussion (20) ¬
- Zulfiqar SheikhWeblinks for ‘DefCon’ at https://www.defcon.org/html/links/defcon-media-archives.html and for ‘FIRST’ (Military aggression database) are not working. Please check these out.
- ZulfiqarThanx 4 ealier info. Bureau of Justice Statistics link (http://bjs.ojp.usdoj.gov/welcome.html) is also broken
-
Thanks for listing the Academic Index on your page :
http://deep-web.org/how-to-research/deep-web-search-engines/
Your evaluation: “Academic Index – ties into deep databases allowed to Texas librarians and educators. A weaker result set than Zuula or Google in my humble opinion.“
I’m not familiar with Zuula, but the whole point of Academic Index is that it is not Google. Academic Index includes only web resources recommend by librarians and teachers world-wide, not just Texas.
We look at websites maintained by teachers and librarians world-wide to find the resources they recommend. We further evaluate the sites we find, and if they are a valuable addition to the search engine, we add them.
We have recently expanded and redesigned the site, and we hope you will be able to take another look at it.
Sincerely,
Dr. Michael Bell- adminOctober 2, 2012, 11:29 pm | ReplyYes, Dr Bell, my mistake. Academic Index was judged against criteria of a meta search engine. Given the nature of the filtered search it produces, I will re-review it and move it into its proper category. I also confirm a change to the index page on 9/11/12 7:49am.
-
Please consider adding vertical deep web file search engine : http://www.findthatfile.com, searches almost all filetypes and protocols and also looks in contents, extracts, and metadata.
-
Greetings
Deep Blog might be useful as well http://deepblog.com/index.html
Best wishes in your thoughtful endevour
Jozef- adminMarch 7, 2013, 6:41 pm | ReplyNot quite a deep web resource, but it does have some good links. A resource would be some database interface to find other things.
-
Very helpful article. I’ve found some sources for searching that I knew nothing about! We all get lulled to thinking that the big G is king, but not when the dark net is involved.Thanks.
-
My input: http://www.findthatfile.com, a file based search engine that searches almost every major source and file type from over 300 million urls.
- Remy BaccinoExcellent article, will help me a lot. How about this site : http://www.scirus.com, for scientific/academic documents and papers search?
- nadiaCIA Factbook — Reference materials containing information on every country in the world.
^ bad link -
Internet Sacred Text Archive (‘ISTA’), Sacred-texts.com
The largest freely available archive of online books about religion, mythology, folklore and the esoteric on the Internet. The site is dedicated to religious tolerance and scholarship, and has the largest readership of any similar site on the web.
Open Source for the Human Soul - Shawn AllenSo I am new, so bare with me, again i need to locate someone urgently, I’ve used the PIPI, people search and intellus, from the surface web and none of them could find current address or phone. I know that what little info thats out there on this person is going to be at the DMV, insurance carrier and post office and W-2s, So anybody got any suggestions ?
thank you - DavidFantastic list. You should add http://SocialCatfish.com. They’re help people find and verify people using images, professional licenses, emails, phone numbers, etc, etc.
- jamesIt appears that Yippy.com is a bad link.
Thanks for the great website. Awesome.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.