This document describes the usage policies and environment of the AMNH Research webservers: research.amnh.org and research-staging.amnh.org. The research department webservers are maintained jointly by the Scientific Web Applications group of the Science Divisions and by the Systems Administration group of IT, on behalf of all of the science departments at the Museum. Research Informatics handles the applications (webserver, database, and so on) and Systems handles most of the system administration (operating system, hardware, etc).
Scientific Web Applications does not administer the public webservers (www, etc.), mailservers, filesharing servers, or any other systems that are not for serving web content. In the case of accessing the servers from outside of the Museum network, however, please note that you need to contact the IT Department Call Center.
Another realm of responsiblity pertaining specifically to networking an network security is that of Network Systems. If you need an an account on the AMNH network (for example, in the case of a new employee or associate) you need to get your department's approval and file a ticket with the Call Center so that Network Systems can grant you access to the network. Likewise, there are some topics, such as domain registration and security, that are matters for Network Systems.
So, in general:
This document defines the policies and instructions for the use of the Research webservers.
Any problems with the Research sites not working, either through unavailability or through software error, should be reported via our contact form. Please use the "report an error" category.
For advice, consulting, or requests for database or website projects, please contact us. Please try to find the answers to your questions in this document before emailing or calling for help.
research.amnh.org is the AMNH science departments' webserver. In addition to research.amnh.org, there is a staging webserver (research-staging.amnh.org) where site updates can be prototyped before they are copied to the production server. All website development is done on the staging server.
research.amnh.org is the production webserver for the Research department websites. It is actually a cluster of two redundant servers. A number of websites are virtually hosted on this cluster. For example, research.amnh.org, www.haydenplanetarium.org, and library.amnh.org all reside here.
Personal websites have URLs on this server, like http://research.amnh.org/~user/, where instead of "user" you substitute the user name. The personal site maintainer actually uploads files to the staging server, research-staging.amnh.org, and they instantly show up on the live server, research.amnh.org. Users should give people URLs to their sites using research.amnh.org, not research-staging. See the section on personal websites for more information.
Any new domain names that should be served on the Research departments' server should be registered as aliases to research.amnh.org.
research-staging.amnh.org is the staging webserver for the Research department sites. A staging server allows you to test changes you want to make to your site, without the public seeing, and without impacting the performance of the live sites in the case of database and cgi applications. Once changes are approved, the prototype files may be published to the live server, using a simple command from the command-line. Another advantage of the two-server system is that, since users need only log into the staging server, security risks on the live server are reduced.
Most of the virtual hosts mentioned in the "live server" section also have their equivalent staging hostnames.
Users who have personal websites on research.amnh.org log into the staging server, research-staging.amnh.org, to make changes to their sites. They do not need to run the webcp program or perform any other extra work to get their content to publish in the way that the departmental site developers do. Any changes made to personal websites will be automatically visible on the live server.
If a user named "someuser" has a personal website, the URL is http://research.amnh.org/users/someuser/and and the directory on the server where the user uploads the files is /v1/public_html/someuser/
We have more documentation available on getting the account that you will need for maintaining a personal site. You may also read about the software you'll need for transferring files, and where to put your files on the server.
Users who maintain science department websites do all of their development work on the staging server. When site revisions are deemed fit for public consumption, the altered files are published to the production webserver from the staging server. See the staging server section for a full discussion of the staging server's role.
There should be one user for each site who has the authority to approve and publish files to the live webserver. Users with publishing authority need to have a shell account on the staging server, and must use Secure Shell (SSH) to access it. See the webcp section for information on webcp, the program used for publishing.
This section describes how research website developers may update files, databases, and personal web pages, and how they may log in to the staging webserver to do programming and to publish their sites. See the server roles section for more information on the webservers' roles.
Once you have determined from reading the section on user scenarios that you need a website account, and once you have read the rest of the current section, please obtain permission from the head of your department or project, and have this person contact the IT Department Call Center to have your account set up.
You will use your account to transfer files from your computer to the webserver. If you simply need to update your personal website, you will put your files in the webserver directory under /v1/public_html that is named after your user name. If you need to update your department's website, see the sections on web development and the filesystem.
A "shell account" is the name for a user account on a Linux server like the Research staging webserver. The "shell" is, specifically, the "command line" interface that a user uses within a terminal program to type commands on the server.
One needs a shell account to publish a departmental site. The shell account may also come in handy for database administration and certain development tasks.
Shell account users need to use the Secure Shell (SSH) protocol (not telnet) for logging in to the staging server (research-staging.amnh.org). Telnet is an insecure protocol that is unencrypted and allows malicious parties on remote servers across the Internet to snoop its connections and obtain users' passwords. Secure shell, on the other hand, is strongly encrypted, and the probability that someone else will capture your password and unencrypt it is extremely slim.
Secure shell also has relatives named Secure Copy (SCP) and SFTP. See below.
Free SSH client software is available for all operating systems:
Here is an example of using SSH from a terminal window (Mac OS X, Linux, etc). Suppose my user name is "someuser" and I want to log in to research-staging.amnh.org. I would open a terminal (e.g. Terminal on a Mac or maybe xterm on Linux) and then type:
ssh someuser@research-staging.amnh.org
With PuTTY or any other GUI client (e.g. for Windows) you just need to make sure you are using the correct host name (research-staging.amnh.org), user name, and password. The port number should be 22, which is probably the default. You can probably leave any other defaults where they are.
The first time you log in to a server, you will get a message about an unrecognized key from the server. This is normal, and you can accept it and proceed. After that, you will be logged in to your shell account on the webserver.
FTP over SSL (FTPS): FTP is a protocol for transferring files over a network — you are probably already familiar with it, or have at least heard of it. FTP over SSL is simply a means of doing FTP over a secured, encrypted connection. After the act of logging in, the experience of using FTPS is the same as with FTP.
The AMNH Research webserver uses explicit FTPS with a clear data channel. This means that it operates over the same network ports as FTP and it uses SSL to encrypt your password when it is sent to the server. There is no slowdown for the actual file transfers because the data-transfer part of the session is not encrypted.
One of the nice things about FTPS is that you are probably already using an FTP program that supports it. Current versions of some of the most popular clients (e.g. WS_FTP, Fetch, Cyberduck, SmartFTP) support it.
There are a number of free and commercial FTP client programs that support FTPS. Here are some free ones:
And here are some commercial ones:
This is not an exhaustive list, but there are more options here.
We have tested FTP/SSL access with SmartFTP and Cyberduck, with success. Both of these are available free of charge.
Settings to use with FTP over SSL:
SSH File Transfer Protocol (SFTP) and Secure Copy (SCP) are related programs for encrypted communications between networked computers. SFTP and SCP are actually related to SSH and are often packaged together. They allow you to log in to a server computer and type commands (SSH) or to transfer files between computers (SCP). If you are familiar already with telnet and FTP, you may think of them as replacements for these protocols. (Technically, they are replacements for rsh and rcp, if you've heard of those.) Additionally, there is a program called rsync that can serve as a wrapper for scp to replicate entire directory trees between computers.
SSH, like telnet, lets you log into a remote computer and execute commands. The difference, though, is that everything gets encrypted, including your password. That way, if someone somewhere on the internet is eavesdropping on your connection they probably won't be able to steal your password and exploit it by breaking into the server. SCP lets you transfer files, with the same benefit. Finally, SFTP is an interactive program like FTP that works over an SSH connection.
For a list of client programs that you can use for SFTP, see above under "FTP over SSL." All of those mentioned above also support SFTP. In addition, you may want to try WinSCP (Windows), or Fugu (Mac) which support SFTP but not FTP over SSL.
The staging websites are accessible inside and outside of the AMNH network. You will be prompted for a username and password for some of the sites if you are browsing them from outside. Please contact us for the user name and password. Also, please note that this does not have anything to do with any real user account, and is in addition to whatever passwords you may have for authenticated areas of certain specific sites.
A database account on research-staging.amnh.org is only necessary if you are developing database applications for your department's site. Please contact us if you need to work with the dbms on research-staging.
The staging server is accessible for file transfers, shell sessions, and web browsing from anywhere on the Internet.
Another option for maintaining your site from home, on the road, or at another institution is a Virtual Private Network (VPN) account. With VPN software, your remote computer is virtually put on the Museum's internal network over an encrypted tunnel, allowing you access to resources that you would not have from the outside. To get VPN access, please see the IT Department's web pages on this topic, and follow the instructions that you find there.
With a VPN (Virtual Private Network) account, you first run a program on your computer that puts your computer virtually on the Museum's internal network, then you connect via one of the protocols described elsewhere in this document.
Please note that, while Research Informatics is in charge of the research webservers, IT is in charge of network access to the server, in other words, everything that happens between your computer and the server.
Your research webserver account is managed by the institutional LDAP system, so you need to change your password by logging into the AMNH's LDAP web interface.
Departmental and project website directories are found under under the directory /srv/www on the staging server. For example, all research.amnh.org files are under /srv/www/research. Within each website directory, you will find a directory named "docs''. This is where website documents go.
For those departmental sites that have their own domain names (for example, library.amnh.org), there is a directory under /srv/www for that site. Thus, for example, library.amnh.org's files are in /srv/www/library.
Personal website directories are in /v1/public_html. Thus, if my user name is "someuser", my personal website directory is /v1/public_html/someuser. See the "personal website" section for more information.
Your user account's home directory is under /home. For example, if my user name is "someuser", my home directory is /home/someuser. This directory is not for storing website files. It is only a small storage area for storing temporary work files and user preference data, and is limited by a filesystem disk quota. Anything significant needs to go in your website directories.
As described in the server roles section, there are two webservers for the Museum's Research websites: a "live'', or publicly-available one, and a staging webserver, for prototyping changes to the sites. After changes have been made, viewed, and approved of, the new or altered files must be replicated to the live server and made public. This is what the webcp program is for.
webcp is run from a command line on the staging server. Only users with the privilege of publishing their site are able to run it, and they are able to publish select directories that are under their control. (You do not have to worry about someone else publishing your prototype content to the world.) Sometimes webcp ability is limited to a designated site maintainer, or a senior department member. For information on gaining publishing privileges, please see the section on getting an account.
Invoke webcp with the following syntax:
shell-prompt$ webcp <directory to publish>or
shell-prompt$ webcp <file to publish>In this example, <directory to publish> would be a web directory on the server's filesystem, like /srv/www/research/docs. Once you run webcp, you will need to wait until the replication finishes. For large directories where a lot has changed, this could take several minutes.
Here is an example:
shell-prompt$ webcp /srv/www/research/docs/my/site/... would take files in the staging website http://research-staging.amnh.org/my/site/ and publish them to the live server at http://research.amnh.org/my/site/.
You can also specify a file name instead of a directory. If you only need to publish a single file, this may be a lot faster than publishing the entire directory in which it resides.
If you receive errors that say, "permission denied'' or "send_files failed to open,'' the cause is probably that the directory you wish to copy is not world-readable, and you should investigate its filesystem permissions.
If you need a new research-related website and do not want it hosted on research.amnh.org, you may either use an alias on the amnh.org domain, or register a new domain for the site, and have that hosted on the Research webserver. An example of the former case would be newsite.amnh.org, and of the latter, www.newsite.org. Setting up a new hostname on amnh.org is easier and faster.
research.amnh.org is the live webserver for the Museum's Research-related sites. Any new Research department domain names or hostnames should be registered as aliases to research.amnh.org.
New domain names that are going to be hosted on Museum webservers should be registered with Network Systems as the administrative and technical contacts for the domain. Specifically, and at the very least, the technical contact email should be hostmaster@amnh.org. This makes it easier for them to be maintained.
We use ns-ext-1.amnh.org as the primary nameserver, and ns-ext-2.amnh.org as the secondary nameserver if you register a domain name (e.g. newsite.org). For example, in the information submitted to the registrar, these hostnames are used for the primary and secondary nameservers. Note that nothing needs to be registered with these organizations if there is just going to be a new host on the amnh.org domain, such as newsite.amnh.org.
Please contact us before you take any of the steps to arrange for a new domain or virtual host. The information above should let you know what to expect, but please contact us to let us help you put everything together.
Please contact us with web programming policy questions.
This section describes resources available to Research department site developers. Feel free to contact us if any of this is unclear.
The W3C maintains an HTML Validation service that you can use to validate the HTML on your site. They also have a CSS Validator. You can check the staging version of your site before you publish it. If your site is one of those that is protected by a password, the W3C Validator will ask you for this password before continuing with its check.
The Drupal content management system, which powers this site, is available to AMNH science websites. Please contact us for more information.
Q: I'm using WinSCP. I'm trying to overwrite a file on the server that I'm supposed to have write access to, but I get an error that says "set mode: not owner'' or some such thing. What can I do?
A: When you copy the file with the buttons at the bottom of the application window, a dialog box will come up with a text box for the destination directory, and three buttons on the bottom: "Copy,'' "Cancel,'' and "More.'' Click the "More'' button and de-select "set permissions'' and "preserve timestamp.''
If you have an account on research-staging.amnh.org, all you need is a directory in /v1/public_html. If you do not have a directory there, or are not sure how to find out, read the sections on personal website users and access policies, and then have the person in charge of your department or project contact us.
Please note that websites for individual researchers are for Museum-related scientific research only, and are not available to assistants and postdoctoral students.
First, make sure you have an account on research-staging. If you do have an account, you should be able to log in via SSH (see the shell account section). Contact us if this does not work.
Once you have an account, see the section on file-transfer methods. You can use SFTP, SCP, or FTP/SSL (FTPS) to upload files.
See "where your files go" to verify that you're trying to put your files in the right directory on the server.
First, check to make sure that you'r really trying to log in to the staging server and not the production server. If you really have the server name right, you probably do not have an account on the server, or you are not using ssh. Note that you may be registered in the LDAP system, but you may not have the particular setting enabled to allow you to log in to this particular server. See the section on getting an account. Also read the rest of this document to get familiar with the system.
You probably don't have filesystem permissions to overwrite or create files in that location. Double-check the name of the directory to which you are trying to upload, and make sure that, if this is a directory that you are sharing with another user, that the other user has set the directory's permissions to allow you to put files there.
You are probably not registered to use webcp, or you do not have permission to publish the particular directory you are trying. Contact us.