Persbericht

VLT Data Flow System Begins Operation

Building a Terabyte Archive at the ESO Headquarters

21 juni 1999

The site at Cerro Paranal in the dry Atacama desert in Northern Chile is one of the best locations for astronomical observations from the surface of the Earth. Each of the four 8.2-m telescopes is a technological marvel with self-adjusting optics placed in a gigantic mechanical structure of the utmost precision, continuously controlled by advanced soft- and hardware. A multitude of extremely complex instruments with sensitive detectors capture the faint light from distant objects in the Universe and record the digital data fast and efficiently as images and spectra, with a minimum of induced noise. And now the next crucial link in this chain is in place. A few nights ago, following an extended test period, the VLT Data Flow System began providing the astronomers with a steady stream of high-quality, calibrated image and spectral data, ready to be interpreted. The VLT project has entered into a new phase with a larger degree of automation. Indeed, the first 8.2-m Unit Telescope, ANTU, with the FORS1 and ISAAC instruments, has now become a true astronomy machine.

A smooth flow of data through the entire system

Already from the start of this project more than ten years ago, the ESO Very Large Telescope was conceived as a complex digital facility to explore the Universe. In order for astronomers to be able to use this marvellous research tool in the most efficient manner possible, the VLT computer software and hardware systems must guarantee a smooth flow of scientific information through the entire system.

This process starts when the astronomers submit well-considered proposals for observing time and it ends with large volumes of valuable astronomical data being distributed to the international astronomical community. For this, ESO has produced an integrated collection of software and hardware, known as the VLT Data Flow System (DFS) , that manages and facilitates the flow of scientific information within the VLT Observatory. Early information about this new concept was published as eso9612 and extensive tests were first carried out at ESOs 3.5-m New Technology Telescope (NTT) at La Silla, cf. eso9703 [1].

The VLT DFS is a complete (end-to-end) system that guarantees the highest data quality by optimization of the observing process and repeated checks that identify and eliminate any problems. It also introduces automatic calibration of the data, i.e. the removal of external effects introduced by the atmospheric conditions at the time of the observations, as well as the momentary state of the telescope and the instruments.

From Proposals to Observations

In order to obtain observing time with ESO telescopes, also with the VLT, astronomers must submit a detailed observing proposal to the ESO Observing Programmes Committee (OPC) . It meets twice a year and ranks the proposals according to scientific merit. More than 1000 proposals are submitted each year, mostly by astronomers from the ESO members states and Chile; the competition is fierce and only a fraction of the total demand for observing time can be fulfilled.

During the submission of observing proposals, DFS software tools available over the World Wide Web enable the astronomers to simulate their proposed observations and provide accurate estimates of the amount of telescope time they will need to complete their particular scientific programme.

Once the proposals have been reviewed by the OPC and telescope time is awarded by the ESO management according to the recommendation by this Committee, the successful astronomers begin to assemble detailed descriptions of their intended observations (e.g. position in the sky, time and duration of the observation, the instrument mode, etc.) in the form of computer files called Observation Blocks (OBs) . The software to make OBs is distributed by ESO and used by the astronomers at their home institutions to design their observing programs well before the observations are scheduled at the telescope.

The OBs can then be directly executed by the VLT and result in an increased efficiency in the collection of raw data (images, spectra) from the science instruments on the VLT. The activation (execution) of OBs can be done by the astronomer at the telescope on a particular set of dates (visitor mode operation) or it can be done by ESO science operations astronomers at times which are optimally suited for the particular scientific programme (service mode operation).

An enormous VLT Data Archive

The raw data generated at the telescope are stored by an archive system that sends these data regularly back to ESO headquarters in Garching (Germany) in the form of CD and DVD ROM disks. While the well-known Compact Disks (CD ROMs) store about 600 Megabytes (600,000,000 bytes) each, the new Digital Versatile Disks (DVD ROMs) - of the same physical size - can store up 3.9 Gigabytes (3,900,000,000 bytes) each, or over 6 times more.

The VLT will eventually produce more than 20 Gigabytes (20,000,000,000 bytes) of astronomical data every night, corresponding to about 10 million pages of text [2]. Some of these data also pass through "software pipelines" that automatically remove the instrumental effects on the data and deliver data products to the astronomer that can more readily be turned into scientific results.

Ultimately these data are stored in a permanent Science Archive Facility at ESO headquarters which is jointly operated by ESO and the Space Telescope European Coordinating Facility (ST-ECF). From here, data are distributed to astronomers on CD ROMs and over the World Wide Web. The archive facility is being developed to enable astronomers to "mine" the large volumes of data that will be collected from the VLT in the coming years. Within the first five years of operations the VLT is expected to produce around 100 Terabytes (100,000,000,000,000 bytes) of data.

It is difficult to visualize this enormous amount of information. However, it corresponds to the content of 50 million books of 1000 pages each; they would occupy some 2,500 kilometres of bookshelves!

The VLT Data Flow System enters into operation

Science operations with the first VLT 8.2-m telescope (ANTU) began on April 1, 1999. Following the first call for proposals to use the VLT in October 1998, the OPC met in December and the observing schedule was finalized early 1999. The related Observation Blocks were prepared by the astronomers in February and March. Service-mode observations began in April and by late May the first scientific programs conducted by ESO science operations were completed.

Raw data, instrument calibration information and the products of pipeline processing from these programs have now been assembled and packed onto CD ROMs by ESO science operations staff. On June 15 the first CD ROMs were delivered to astronomers in the ESO community.

This event marks the closing of the data flow loop at the VLT for the first time and the successful culmination of more than 5 years of hard work by ESO engineers and scientists to implement a system for efficient and effective scientific data flow. This was achieved by a cross-organization science operations team involving staff in Chile and Europe.

With the VLT Data Flow System, a wider research community will have access to the enormous wealth of data from the VLT. It will help astronomers to keep pace with the new technologies and extensive capabilities of the VLT and so obtain world-first scientific results and new insights into the universe.

Noten

[1] A more technical description of the VLT Data Flow System is available in Chapter 10 of the VLT Whitebook.

[2] By definition, one "normal printed page" contains 2,000 characters.

Connect with ESO on social media

Over dit bericht

Persberichten nr.:eso9930
Legacy ID:PR 10/99
Naam:ESO Data Centre
Type:Unspecified : Technology : Observatory : Facility
Unspecified : Technology : Observatory : Telescope
Facility:Very Large Telescope

Afbeeldingen

The VLT Data Flow System
The VLT Data Flow System
First DVD storage robot for the VLT data archive
First DVD storage robot for the VLT data archive
Working with the VLT data archive
Working with the VLT data archive

Send us your comments!
Abonneer op ESO-nieuws in uw eigen taal
Accelerated by CDN77
Terms & Conditions
Cookie Settings and Policy

Our use of Cookies

We use cookies that are essential for accessing our websites and using our services. We also use cookies to analyse, measure and improve our websites’ performance, to enable content sharing via social media and to display media content hosted on third-party platforms.

You can manage your cookie preferences and find out more by visiting 'Cookie Settings and Policy'.

ESO Cookies Policy


The European Organisation for Astronomical Research in the Southern Hemisphere (ESO) is the pre-eminent intergovernmental science and technology organisation in astronomy. It carries out an ambitious programme focused on the design, construction and operation of powerful ground-based observing facilities for astronomy.

This Cookies Policy is intended to provide clarity by outlining the cookies used on the ESO public websites, their functions, the options you have for controlling them, and the ways you can contact us for additional details.

What are cookies?

Cookies are small pieces of data stored on your device by websites you visit. They serve various purposes, such as remembering login credentials and preferences and enhance your browsing experience.

Categories of cookies we use

Essential cookies (always active): These cookies are strictly necessary for the proper functioning of our website. Without these cookies, the website cannot operate correctly, and certain services, such as logging in or accessing secure areas, may not be available; because they are essential for the website’s operation, they cannot be disabled.

Cookie ID/Name
Description/Purpose
Provider (1st party or 3rd party)
Browser session cookie or Stored cookie?
Duration
csrftoken
XSRF protection token. We use this cookie to protect against cross-site request forgery attacks.
1st party
Stored
1 year
user_privacy
Your privacy choices. We use this cookie to save your privacy preferences.
1st party
Stored
6 months
_grecaptcha
We use reCAPTCHA to protect our forms against spam and abuse. reCAPTCHA sets a necessary cookie when executed for the purpose of providing its risk analysis. We use www.recaptcha.net instead of www.google.com in order to avoid unnecessary cookies from Google.
3rd party
Stored
6 months

Functional Cookies: These cookies enhance your browsing experience by enabling additional features and personalization, such as remembering your preferences and settings. While not strictly necessary for the website to function, they improve usability and convenience; these cookies are only placed if you provide your consent.

Cookie ID/Name
Description/Purpose
Provider (1st party or 3rd party)
Browser session cookie or Stored cookie?
Duration
Settings
preferred_language
Language settings. We use this cookie to remember your preferred language settings.
1st party
Stored
1 year
ON | OFF
sessionid
ESO Shop. We use this cookie to store your session information on the ESO Shop. This is just an identifier which is used on the server in order to allow you to purchase items in our shop.
1st party
Stored
2 weeks
ON | OFF

Analytics cookies: These cookies collect information about how visitors interact with our website, such as which pages are visited most often and how users navigate the site. This data helps us improve website performance, optimize content, and enhance the user experience; these cookies are only placed if you provide your consent. We use the following analytics cookies.

Matomo Cookies:

This website uses Matomo (formerly Piwik), an open source software which enables the statistical analysis of website visits. Matomo uses cookies (text files) which are saved on your computer and which allow us to analyze how you use our website. The website user information generated by the cookies will only be saved on the servers of our IT Department. We use this information to analyze www.eso.org visits and to prepare reports on website activities. These data will not be disclosed to third parties.

On behalf of ESO, Matomo will use this information for the purpose of evaluating your use of the website, compiling reports on website activity and providing other services relating to website activity and internet usage.

ON | OFF

Matomo cookies settings:

Cookie ID/Name
Description/Purpose
Provider (1st party or 3rd party)
Browser session cookie or Stored cookie?
Duration
Settings
_pk_id
Stores a unique visitor ID.
1st party
Stored
13 months
_pk_ses
Session cookie temporarily stores data for the visit.
1st party
Stored
30 minutes
_pk_ref
Stores attribution information (the referrer that brought the visitor to the website).
1st party
Stored
6 months
_pk_testcookie
Temporary cookie to check if a visitor’s browser supports cookies (set in Internet Explorer only).
1st party
Stored
Temporary cookie that expires almost immediately after being set.

Additional Third-party cookies on ESO websites: some of our pages display content from external providers, e.g. YouTube.

Such third-party services are outside of ESO control and may, at any time, change their terms of service, use of cookies, etc.

YouTube: Some videos on the ESO website are embedded from ESO’s official YouTube channel. We have enabled YouTube’s privacy-enhanced mode, meaning that no cookies are set unless the user actively clicks on the video to play it. Additionally, in this mode, YouTube does not store any personally identifiable cookie data for embedded video playbacks. For more details, please refer to YouTube’s embedding videos information page.

Cookies can also be classified based on the following elements.

Regarding the domain, there are:

As for their duration, cookies can be:

How to manage cookies

Cookie settings: You can modify your cookie choices for the ESO webpages at any time by clicking on the link Cookie settings at the bottom of any page.

In your browser: If you wish to delete cookies or instruct your browser to delete or block cookies by default, please visit the help pages of your browser:

Please be aware that if you delete or decline cookies, certain functionalities of our website may be not be available and your browsing experience may be affected.

You can set most browsers to prevent any cookies being placed on your device, but you may then have to manually adjust some preferences every time you visit a site/page. And some services and functionalities may not work properly at all (e.g. profile logging-in, shop check out).

Updates to the ESO Cookies Policy

The ESO Cookies Policy may be subject to future updates, which will be made available on this page.

Additional information

For any queries related to cookies, please contact: pdprATesoDOTorg.

As ESO public webpages are managed by our Department of Communication, your questions will be dealt with the support of the said Department.