La Silla/Paranal Unification Project
SciOps:
General Considerations
|
Prepared: ohainaut
Version: 2003-09-29 ohainaut
Introduction & Purpose
Several scenarii can be considered for the unification of SciOpses:
-
-a- We don't do anything, the merge is just a change of name, but both SciOpses continue to operate as now.
-
-b- The operation scheme of Paranal is directly applied to La Silla.
-
-c- The operation scheme of La Silla is directly applied to Paranal
-
-d- The new operation scheme is a merge of the current ones
-
-e- The new, unified operation scheme is the result of
a re-engineering that not only consider the current
schemes, but explore new ones.
a- and c- are listed for completeness; b- is described in details in
[1]
In my view, our duty is to improve the operation of both
observatories by a full re-engineering, i.e. -e-, including changes
-hopefully improvements- to both LSO and PAO operation schemes.
Moreover, this is a unique opportunity to get what we have always
dreamed for, since, if we decide that we need something for the merge to work, it is likely that we will get it.
What follows is not a full implementation of an operation plan, but
some ideas that should be explored/discussed/expanded/destroyed in the
process of re-engineering.
Relevant Documents:
[1] La Silla - Paranal unification - gmathys
1- Industrial data/ crafted data
The PAO has been compared to an "industrial backery", producing a
high quality AND highly standardized product (data, not croissants). On the other hand, LSO
has been compared to an old fashioned, artesanal backery, producing
also high quality product, but that could be tuned to the exact
requirements of a client when these are not standard. I suggest to
formalize and expand this:
- Restrict the "hand crafted" data mode to highly ranked
programs that push the instrument beyond its standard modes.
For
these programs, concepts like shutter efficiency should be of no
relevance, since these aim at quality, not quantity. Obviously, these
programs should be restricted to a fraction of the time (to be decided
by relevant committees), but a significant fraction of the Visitor Mode
time, and possibly a small fraction of Service Mode time could be a
starting point.
In this mode, whatever support needed by the programs
has to be allocated; from the SciOp point of view, this means full time
availability of one of the instrument top specialist, and one TIO.
- For programs in "Industrial Mode", i.e. programs for which quantity and overall
quality are more important than the quality of each and every
individual frame, apply modern
industrial concepts. This implies -as a start-
- Full automatization of all
repetitive tasks, totally eliminate burden of repetitive operation to
strongly qualified staff (either by automatization or using
appropriately qualified technicians, as opposed to PhD/Engineer): a
repetitive task will be --on average-- performed better and much more
uniformly than a collection of human.
- Implementation/improvement of appropriate tools to permit the execution in quasi automatic mode of the observations: eg tools to
prepare observing queues, execute them, classify these data in a very efficient way.
- Definition of quality parameters that we would guarantee.
For instance, a minimum of xx% (TBD) of observations declared "successfull"
actually reaching the requested quality, implying that up to 2% of
losses (data to be trashed) can be considered as acceptable. Indeed, 100% success would require
much higher cost (basically similar to those of the "hand crafted"
data).
In "industrial production" mode, a telescope can be
run almost in automatic mode, one TIO driving the whole system, with (a
fraction of) an astronomer available for assessing complex cases
(complex change of service queue, difficult acquisition, strange case
of data quality control). This frees a considerable amount of highly
qualified manpower for tasks requiring their qualification.
These two modes should be detailed and explained to the
community, in particular that "crafted data" are definitely not optimal
for a standard mass-production program, and that "industrial data" is
not (necessarily) the same as service mode.
2- Service/Visitor modes:
Various scenarii can be considered:
- No service. Probably not an option, since this mode is highly requested by the community.
- Minimum service (as currently at la Silla): only "micro-programs" and "monitoring programs".
- Queue scheduling: to have a good collection of program
efficiently sampling the various conditions, a significant fraction of
the time must be schedulled in this mode - typically >=50%. This is
the bulk of the "Industrial Production".
- Full service (~100% of the available time): this give the
largest flexibility to the observatory, in terms of schedulling of the
programs and technical time, and of the staff, but has the strongest
impact in terms of loss of contacts between the community and the
observatory; this is discussed below.
Service-related problem: loss of contact between the Community and the Observatory:
In scenarii with a high fraction of Service Mode,
the number of visiting astronomers will strongly decrease. In a first
stage this would result in a increasing Phase-II preparation problems
(PI with unrealistic ideas/expectations), then in a strong problem for
staff replacement, since our potential staff pool is the Community.
As Service Observing is likely to stay, we have to take corrective actions. Possible ideas are listed below.
- Pairs of Visitors ("experienced observer" coming with a
"novice"). This could be "encouraged" for short runs, and "strongly
recommended" for long runs. Alt, it could be encouraged for PaO,
recommended for LSO (where lodging is not an issue).
- Modify the ESO-Studentship program to include training work at
the observatory. This can be done fairly informally (student helping
staff), or formally (with formal training, and a level of formal
duties). This should be designed to be cost-neutral in the short term,
i.e. the time invested in training paid for by direct work provided by
the student. The long-term gain is obvious, as all ESO-student would
have the training to understand the basic operation of the observatory
(and hopefully to give them the taste of it).
- "Summer school" (winter?) programs, where ~10 students come to
work for ~2 weeks, to get formal training on operation/instrumentation
aspects that are not covered in the graduate programs (from technical
aspects proposal writing (instrument selection, ETC, ...), observation
preparation, observation execution and real-time reaction, quality
control, etc...). Obviously, this would have impacts both on the
schedule (since some observing time should be reserved for these
students) and on the staffing level (since the preparation and delivery
of such training would be very demanding).
3- Optimization
Any task that is currently repeated several times (per day, week,
period...) must be automatized or delegated to technical staff
(basically technicians, as opposed to highly qualified/expensive staff
as TIOs, Astronomers, DHI).
All staff should be utilized to perform tasks that require their respective education/expertize. In particular:
TIOs
- ESO TIOs are either very experienced with telescopes/instruments
and/or highly educated engineers. They should be given much more
responsibilities than just driving the TCS. Driving the full Telescope
+ Instrument seems a minimum. Being in charge of the operation of
Telescope + Instrument is certainly within reach after proper training
of the TIOs. Being
"in charge", opposed to simply "driving" includes the first level of
trouble-shooting.
- La Silla TIOs are alternately on Day and Night shift. During the day, the TIO
- executes and checks required day calibrations (CALOB or equivallent)
- performs instrument set-up required (EMMI, EFOSC, CES)
- executes operation maintenance tasks on telescopes and instruments
- starts-up the telescopes and instruments
- coordinates all actions taking place on the telescope (Engineering interventions, maintenance, etc).
A significant benefit of this day work
is that the TIOs get a much deeper knowledge and understanding of the
systems, which allows them to operate in a very efficient way.This makes the front line night time troubleshooting very efficient.
It has also some side advantages on the personal side: the integration
of the TIOs in the observatory is much deeper than when they were
night-only, and breaks the monotony of night-only work.We
should discuss the benefit of both models (night only vs day/night).
Astronomer:
- Service observing / Industrial production:
SM process can be divided in various steps:
- Phase II - currently done by USG for PAO and WFI, LS-SciOp for
the others. Should be fully passed to USG in order to eliminate
duplication of efforts and more uniformity for the PIs. Cost to USG has
to be estimated.
- MTS: same as Ph-II.
- STS: currently
done "by hand" by SciOp Night Astro, using input from MTS, and Day
astronomer (at PAO) and various personal tools, tricks and tips. In
particular, the existing supporting tools are still fairly
primitive (eg. does not consider
Moon constraints and time critical observations). A strong development
in the STS, taking into account all available constraints (inc. Moon
and time critical) as well as "present" conditions (seeing, wind, moon,
clouds) is required, so that the preliminary queues can be update
instantly at any time. With the addition of "priority rules" (ABC, ToO,
Chile, etc...), and rules to try and minimize the number of
instrumental configuration, the output queue should be close to optimum.
Keep in mind that funy programs are not to be executed in industrial
mode; in a first step, programs with strange time constraints or series
of linked OBs (not yet considered by STS) have to be executed in
"handcraft mode" .
In a second step, the STS should include available
short term predictions.
The SciOp astro can then check/certify
the STS pre-sequences generated in the afternoon (eg pushing in
or removing some programs). The Night astro must be available at
all time for real-time interventions not considered by the STS and/or
to clarify doubts of the TIO.
- Execution: currently, the SciOp astro select the next OBs in
the OT queues, and watch their execution. In many cases, acquisition is
straighforward. The actual execution of the queue should be transfered
to the TIO, who will then have a complete overview of the observation
execution process. The Astronomer must be available at all time for
assistance in case of problems (investigate how many per night we
currently have).
- Quality control
check.
The actual check of the constraints can be fully automatized.
The QC performed at the telescope is
very basic: were the requested conditions satisfied? is the expected
object visible? is the exposure level OK (i.e. not saturated, decent
exposure?). At this stage, there is no further QC performed on the
data, either on- or off-site. Currently, this QC is performed by hand
by the Night Astronomer, and stored in NightLog.
Most of the QC must be automatized: many of the
parameter estimations don't require human intervention (moon distance
and phase are computed, seeing/strehl measured by the pipeline, cloud
coverage estimated by All-Sky cam). Ideally/ultimately, Pipeline-based
QC should ensure the required quality (98%); in the
mean-time.
- Visiting Astronomer / Industrial production
- Pre-introduction (P2PP et al) by the (generic) astro.
- Installation at telescope (day of observations), astro.
- Begin of night, hand-holding till first set(s) of successfull science exposures, astro.
- Routine observations: TIO runs the system, while visitor drives P2PP and works on his data.
- Night astronomer is available to answer questions from the Visitor.
- Handcraft Mode
- Require the presence of one of the instrument specialist for
the preparation (P2PP), introduction and Night Support, basically when
the visitors need him.
- Shift Leader
- The role of the shift leader is to ensure 100% time coverage of
a Senior Astronomer to solve any astronomy related emergency, and more
generally to assist the HoSciOp as a deputy.
- Background work
- Maintenance: general check of instrument health, evolution, etc. Can be minimized tru AutRep/Pipeline/QC
- Web and documentation
- Research and development: this is what astronomers are good at,
and where they should be used as much as possible: development of new
tools for analysis, investigation of in-depth behaviour of the
instruments, etc.
Data Handling Administrator
- DVD production for Archive: this manpower-intensive task should be replaced by NGAS technology asap.
- DVD/CD production for Visitors: quite automatic in Paranal, should be automatized at LSO asap.
- Database operation and administration, and DFS maintenance and
operation should be the main activity of DHA. (how many? what is role
of DHA in DFS maintenance and upgrades)
Operation Engineers
Operation Engineers, or operation managers, exist at La Silla. Their key role involves
- Procedure Certification
In case of procedure change/upgrade, the new procedure has to be
certified by the OpEng. This permits overall control, uniformity and
configuration control
- Configuration Control
The OpEngs are responsible for the configuration control of all
operation-related procedures, documents (esp. check lists and
written/software (OSF) procedures) and SciOp produced software
While some of their tasks could be passed to engineering/software,
their role is very deeply linked to the Operation side of SciOp. We
should investigate their status in the unified scheme (i.e. do we want
OpEng? if not, who does their work? would PAO-SciOp (have) benefit(ted)
to have some "in house" engineering expertise for day-to-day operation
management?
---oOo---