3.5 Ceph

ceph Main.

Before use

3.5.1 Ceph Health Monitoring

We can check the cluster's overall health in real time, and intuitively identify the cause when a problem occurs.

• Full state summary card (Left side.) Currently: Ceph It visualizes the overall state of the cluster in color and icon so that it's easy to see.
  • It's normal. (Healthy) : All services and data copying processes are in a normal state of operation. (Green.)
  • Warning. (Warning) : In the HEALTH_WARN state, the service is running or some OSD space shortage, copy counting, etc. requires caution. (Yellow.)
  • Error (Error) : In a HEALTH_ERR state, there is a risk of data loss or major service interruptions and immediate action is needed. (Red.)
  • Update the scheduleData is automatically updated every 10 seconds, and you can update it automatically with the "Fresh" button at the top.
• The status of the diagnostic inbound. (Right.) It shows a list of specific issues that have occurred. A simple name for the languageIt's converted to.
  • PG (Placement Group) Related status
    • Data copy shortage (Degraded) : PG_DEGRADED state: the number of secure copies of data is the target(Replica count) It's less than that. (OSD down/reboot occurred mainly)
    • The number of copies (Undersized) : PG_UNDERSIZED: We can't find enough OSD to store the data, and we can't make a copy.
    • Regular check-ups are delayed. (Not Deep Scrubbed) PGNOTDEEP_SCRUBBED state Ceph The data is periodically carefully checked for data integrity.(Deep Scrub) But this wasn't done in time.
    • Check-in is underway. (Not Scrubbed) PGNOTSCRUBBED state data integrity check(Scrub) This hasn't been done for a long time, so the system load is too high, and the check might be pushing, so run it manually or reduce the load.
    • Detecting data damage (Damaged) PG_DAMAGE: Serious data damage was detected in the state. Data recovery may be impossible.ceph You have to try to recover with the pg repair command, and if it fails, you have to recover the data from the backup.
    • Data disagreements (Inconsistent) PG_INCONSISTENT: The data content between the replicas is different from each other.
    • Data entry and exit shutdown. (Availability) PG_AVAILABILITY: Some data is inaccessible to some states, and it's very serious, it's impossible to read/write.(Peering) If this is blocked, check the network status.
    • Lack of recovery space. (Recovery Full) PGRECOVERYFULL: You need to go through the data recovery process, and there's no space left in the OSD, so you can't do it. You need to add a new OSD and get the capacity, and the recovery will resume.
  • OSD (Object Storage Daemon) Related status
    • OSD is down. (OSD Down) : OSD_DOWN state of one or more storage demons(OSD) It's either stopped working or the network is disconnected.
    • The shortage of storage space is imminent. (Near Full) OSD_NEARFULL: Cluster storage is filling up. (More than 85% of the basic)
    • It's full of storage space. (Full) : OSD_FULL: The storage space limit has reached, and data writing has stopped. (It's very dangerous.)
    • There's no shortage of space. (Backfill Full) : Rearrange the data to OSD_BACKFILLFULL state(Backfill) And the space was limited to try and make it work.
    • OSD checking error (Scrub Errors) I'm going to check the disk.(Scrub) We found a physical backsector or a reading error during the process.(SMART) Check the information and check the disk swap.
    • The shell is recovering. (Too Many Repairs) OSDTOOMANY_REPAIRS state is that there are too many read errors/corrections occurring in a particular OSD.
    • The disk state. (Device Health) : DEVICEHEALTHWARNING state: In the SMART state of the disk, the problem is detected and the life expectancy is the warning stage.
  • Performance and network latency (Slow Ops / Ping) Related status
    • Delayed processing (Slow Ops) SLOW_OPS: The read/write request processing is taking an unusually long time, which is the main cause of low performance, and the hard drive load of a particular OSD.(The cannon.) Check if the network bandwidth is saturated.
    • Backend network delay (Slow Ping Back) : OSDSLOWPINGTIMEBACK state for internal data replication between OSDs(Cluster) Network response speeds are slow, so check the internal network switches, cable status, or check the network load of a specific node.
    • Frontend network delay (Slow Ping Front) : OSDSLOWPINGTIMEFRONT state to communicate with the client externally(Public) Network response speeds are slow, make sure the service network has enough bandwidth, and that the switch settings are fine.
  • Monitor and Manager related status
    • Monitor down (Monitor Down) : Some of the monitoring nodes that manage cluster maps in MON_DOWN state are not responding. (I need to see if the quorum is maintained.)
    • Time sync error (Clock Skew) : MONCLOCKSKEW state: The system time between the monitor nodes is different from each other. Ceph It's very sensitive to time synchronization.
    • Messenger v2 is disabled. (Msgr2 Not Enabled) The latest communication protocol is in MONMSGR2NOT_ENABLED state(v2) This is not activated.
    • Manager down (Manager Down) MGR_DOWN state is managed(Mgr) All the Demons are dead. The Dashboard statistics are not updated.
    • Manager module error (Mgr Module Error) MGRMODULEERROR is in the state Ceph Manager internal plugins(The Dashboard, the Prometheus, and so on.) There was an error in the
  • Authentication and Setup (Frequent occurrence during upgrades)
    • Security vulnerability connection detection (Insecure Global ID) AUTHINSECUREGLOBALIDRECLAIM: An old client is trying to reconnect in a security-vulnerable way.
    • The demons are detecting the bubble. (Old Version) Some demons are out of state. Ceph I'm running. The upgrade may not have been completed.
    • DB partition capacity is exceeded (BlueFS Spillover) BLUEFS_SPILLOVER: The DB partition for fast speeds is full, and metadata is flooding into slow, general data areas.
  • Normal and other
    • It's normal. : HEALTH_OK state: All services are working normally.
    • Warning. : HEALTH_WARN: Cluster operation is fine, but caution is needed.
    • Error : HEALTH_ERR is a serious condition that requires immediate response.
    • Except for this one. Ceph Please check the status code reported in the cluster for detailed logs.

3.5.2 Ceph The status of the diagnostic in-line detailed modeling window

This is a detailed information window that comes out when you click on a specific diagnostic entry in the list.

• The status of the data: Hangout naming and technical description of the error code(Description) I'll show you at the top.
• Help and Resolution GuideNot just simple error messages, but specific actions that managers should take.(Solution)* is given as a guide box.
• Detailed logs: The Law on the Problems(Raw) And we're going to provide log data in chronological order. Copy the log. You can save it on the clipboard with a button.
• The original message.For a skilled manager. Ceph The original message in the English language that you returned is also marked with it.

3.5.3 Ceph Monitoring of the service status

Ceph It monitors the inventory and status of the core demons that make up the cluster in the form of cards.

• And the common ground is that the service status It's normal. (Active) - (Green.), Wait a minute. (Standby) - (It's gray.), Stop the car. (Stopped) - (Red.)And the part is divided into three parts.
• Monitors (Watch the police.) Cluster map:(Cluster Map) And the quorum.(Quorum) They're in the quorum, acting as leaders or followers.
• Managers (The manager.) : It's responsible for dashboard, monitoring, and metric collection.
  • ActiveCurrent state(Primary) I'm in the role of manager.
  • StandbyHe's waiting to take over the role of the chief executive officer.
• Metadata Servers (MDS) : I manage the metadata of the CephFS file system.
  • ActiveI'm processing the file system request.
  • Standby: Waiting for load distribution or disruption measures.

3.5.4 Ceph Monitoring the state of OSDs

Physical disk where the actual data is stored(OSD) The state of The node.(The server.) Grouping them into units.And we manage it.

• OSD summary card (Upstairs, please.)
  • The total number of OSDs: Total number of OSDs registered in the cluster.
  • It's working normally. (Up): The number of OSDs the process is running normally.
  • Disrupted / Disrupted (Down): The number of OSDs that need to be checked.
  • The capacity is short. (>85%): The number of OSDs whose use has reached the level of risk.
• Detailed list of OSDs (Grouping by node)
  • The physical server.(Node) It's not easy to pinpoint a server's failure by tying the OSD together.
  • IDThe OSD's unique identification number: (Yeah, 0, 1, 2...).
  • It's working. (Status)OSD: Whether or not the demo process is running.
    • It's normal. (Up): Process running (Green.).
    • Stop it. (Down): The process is stopped (Red.).
  • Data placement (In/Out): whether data is allocated by the CRUSH algorithm.
    • Placement (In)I'm participating in the cluster data storage. (Blue.).
    • Except for the (Out): Excluded from data allocation for maintenance and other reasons (It's gray.).
  • Type of (Type)Physical type of disk: (HDD, SSD, NVMe) It's sort of like an icon.
  • The device path. (Device): Block device path on Linux system (Yes, I know. /dev/sda, /dev/dm-0).
  • Use of the (Usage)The total capacity of the disk: The Progress Bar.It visualizes the ro.
    • Blue.Stable: (< 70%)
    • Yellow.Attention: (70% ~ 85%)
    • Red.The risks: (> 85%)
  • PG numberSo, the number of placement groups assigned to the OSD is, is the data being distributed correctly.(Road balancing) We can see.

How to solve problems