Ubuntu Server Guide Changes, errors and bugs
Download 1.27 Mb. Pdf ko'rish
|
ubuntu-server-guide
- Bu sahifa navigatsiya:
- Support and Troubleshooting
|
Tuning Openvswitch-DPDK
DPDK has plenty of options - in combination with Openvswitch-DPDK the two most commonly used are: ovs−v s c t l s e t Open_vSwitch . o t h e r _ c o n f i g : n−dpdk−r x q s=2 ovs−v s c t l s e t Open_vSwitch . o t h e r _ c o n f i g : pmd−cpu−mask=0x6 The first select how many rx queues are to be used for each DPDK interface, while the second controls how many and where to run PMD threads. The example above will utilize two rx queues and run PMD threads on CPU 1 and 2. See the referred links to “EAL Command-line Options” and “OpenVswitch DPDK installation” at the end of this document for more. As usual with tunings you have to know your system and workload really well - so please verify any tunings with workloads matching your real use case. Support and Troubleshooting DPDK is a fast evolving project. In any case of a search for support and further guides it is highly recom- mended to first check if they apply to the current version. Check if it is a known issue on: • DPDK Mailing Lists • For OpenVswitch-DPDK OpenStack Mailing Lists • Known issues in DPDK Launchpad Area • Join the IRC channels #DPDK or #openvswitch on freenode. Issues are often due to missing small details in the general setup. Later on, these missing details cause problems which can be hard to track down to their root cause. A common case seems to be the “could not open network device dpdk0 (No such device)” issue. This occurs rather late when setting up a port in Open vSwitch with DPDK. But the root cause most of the time is very early in the setup and initialization. Here an example how a proper initialization of a device looks - this can be found in the syslog/journal when starting Open vSwitch with DPDK enabled. ovs−c t l [ 3 5 6 0 ] : EAL: PCI d e v i c e 0 0 0 0 : 0 4 : 0 0 . 1 on NUMA s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: probe d r i v e r : 8 0 8 6 : 1 5 2 8 rte_ixgbe_pmd 72 ovs−c t l [ 3 5 6 0 ] : EAL: PCI memory mapped a t 0 x 7 f 2 1 4 0 0 0 0 0 0 0 ovs−c t l [ 3 5 6 0 ] : EAL: PCI memory mapped a t 0 x 7 f 2 1 4 0 2 0 0 0 0 0 If this is missing, either by ignored cards, failed initialization or other reasons, later on there will be no DPDK device to refer to. Unfortunately the logging is spread across syslog/journal and the openvswitch log. To allow some cross checking here an example what can be found in these logs, relative to the entered command. #Note : This l o g was taken with dpdk 2 . 2 and o p e n v s w i t c h 2 . 5 but s t i l l l o o k s q u i t e s i m i l a r ( a b i t extended ) t h e s e days Ca pt io n s : CMD: t h a t you e n t e r SYSLOG: ( I n l c u d i n g EAL and OVS Messages ) OVS−LOG: ( Openvswitch me ssa g es ) #PREPARATION Bind an i n t e r f a c e t o DPDK UIO d r i v e r s , make Hugepages a v a i l a b l e , e n a b l e DPDK on OVS CMD: sudo s e r v i c e openvswitch −s w i t c h r e s t a r t SYSLOG: 2016−01−22T08 : 5 8 : 3 1 . 3 7 2 Z | 0 0 0 0 3 | daemon_unix ( monitor ) | INFO | p i d 3329 died , k i l l e d ( Terminated ) , e x i t i n g 2016−01−22T08 : 5 8 : 3 3 . 3 7 7 Z | 0 0 0 0 2 | v l o g | INFO | opened l o g f i l e / var / l o g / o p e n v s w i t c h / ovs−v s w i t c h d . l o g 2016−01−22T08 : 5 8 : 3 3 . 3 8 1 Z | 0 0 0 0 3 | ovs_numa | INFO | D i s c o v e r e d 12 CPU c o r e s on NUMA node 0 2016−01−22T08 : 5 8 : 3 3 . 3 8 1 Z | 0 0 0 0 4 | ovs_numa | INFO | D i s c o v e r e d 1 NUMA nodes and 12 CPU c o r e s 2016−01−22T08 : 5 8 : 3 3 . 3 8 1 Z | 0 0 0 0 5 | r e c o n n e c t | INFO | unix : / var / run / o p e n v s w i t c h /db . s o c k : c o n n e c t i n g . . . 2016−01−22T08 : 5 8 : 3 3 . 3 8 3 Z | 0 0 0 0 6 | r e c o n n e c t | INFO | unix : / var / run / o p e n v s w i t c h /db . s o c k : c o n n e c t e d 2016−01−22T08 : 5 8 : 3 3 . 3 8 6 Z | 0 0 0 0 7 | b r i d g e | INFO | ovs−v s w i t c h d ( Open vSwitch ) 2 . 5 . 0 OVS−LOG: systemd [ 1 ] : S t o p p i n g Open vSwitch . . . systemd [ 1 ] : Stopped Open vSwitch . systemd [ 1 ] : S t o p p i n g Open vSwitch I n t e r n a l Unit . . . ovs−c t l [ 3 5 4 1 ] : * K i l l i n g ovs−v s w i t c h d ( 3 3 2 9 ) ovs−c t l [ 3 5 4 1 ] : * K i l l i n g ovsdb−s e r v e r ( 3 3 1 8 ) systemd [ 1 ] : Stopped Open vSwitch I n t e r n a l Unit . systemd [ 1 ] : S t a r t i n g Open vSwitch I n t e r n a l Unit . . . ovs−c t l [ 3 5 6 0 ] : * S t a r t i n g ovsdb−s e r v e r ovs−v s c t l : ovs | 0 0 0 0 1 | v s c t l | INFO | C a l l e d a s ovs−v s c t l −−no−w a i t −− i n i t −− s e t Open_vSwitch . db−v e r s i o n = 7 . 1 2 . 1 ovs−v s c t l : ovs | 0 0 0 0 1 | v s c t l | INFO | C a l l e d a s ovs−v s c t l −−no−w a i t s e t Open_vSwitch . ovs−v e r s i o n = 2 . 5 . 0 ” e x t e r n a l −i d s : system−i d =\”e7c5ba80−bb14 −45c1−b8eb −628 f 3 a d 0 3 9 0 3 \”” ” system−type =\”Ubuntu \”” ” system−v e r s i o n =\”16.04 − x e n i a l \”” ovs−c t l [ 3 5 6 0 ] : * C o n f i g u r i n g Open vSwitch system IDs ovs−c t l [ 3 5 6 0 ] : 2016−01−22T08 : 5 8 : 3 1 Z | 0 0 0 0 1 | dpdk | INFO | No −vhost_sock_dir p r o v i d e d − d e f a u l t i n g t o / var / run / o p e n v s w i t c h ovs−v s w i t c h d : ovs | 0 0 0 0 1 | dpdk | INFO | No −vhost_sock_dir p r o v i d e d − d e f a u l t i n g t o / var / run / o p e n v s w i t c h 73 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 0 a s c o r e 0 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 1 a s c o r e 1 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 2 a s c o r e 2 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 3 a s c o r e 3 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 4 a s c o r e 4 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 5 a s c o r e 5 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 6 a s c o r e 0 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 7 a s c o r e 1 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 8 a s c o r e 2 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 9 a s c o r e 3 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 10 a s c o r e 4 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d l c o r e 11 a s c o r e 5 on s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: Support maximum 128 l o g i c a l c o r e ( s ) by c o n f i g u r a t i o n . ovs−c t l [ 3 5 6 0 ] : EAL: D e t e c t e d 12 l c o r e ( s ) ovs−c t l [ 3 5 6 0 ] : EAL: VFIO modules not a l l loaded , s k i p VFIO s u p p o r t . . . ovs−c t l [ 3 5 6 0 ] : EAL: S e t t i n g up p h y s i c a l l y c o n t i g u o u s memory . . . ovs−c t l [ 3 5 6 0 ] : EAL: Ask a v i r t u a l a r e a o f 0 x100000000 b y t e s ovs−c t l [ 3 5 6 0 ] : EAL: V i r t u a l a r e a found a t 0 x 7 f 2 0 4 0 0 0 0 0 0 0 ( s i z e = 0 x100000000 ) ovs−c t l [ 3 5 6 0 ] : EAL: R e q u e s t i n g 4 p a g e s o f s i z e 1024MB from s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: TSC f r e q u e n c y i s ~2397202 KHz ovs−v s w i t c h d [ 3 5 9 2 ] : EAL: TSC f r e q u e n c y i s ~2397202 KHz ovs−v s w i t c h d [ 3 5 9 2 ] : EAL: Master l c o r e 0 i s ready ( t i d=f c 6 c b b 0 0 ; c p u s e t = [ 0 ] ) ovs−v s w i t c h d [ 3 5 9 2 ] : EAL: PCI d e v i c e 0 0 0 0 : 0 4 : 0 0 . 0 on NUMA s o c k e t 0 ovs−v s w i t c h d [ 3 5 9 2 ] : EAL: probe d r i v e r : 8 0 8 6 : 1 5 2 8 rte_ixgbe_pmd ovs−v s w i t c h d [ 3 5 9 2 ] : EAL: Not managed by a s u p p o r t e d k e r n e l d r i v e r , s k i p p e d ovs−v s w i t c h d [ 3 5 9 2 ] : EAL: PCI d e v i c e 0 0 0 0 : 0 4 : 0 0 . 1 on NUMA s o c k e t 0 ovs−v s w i t c h d [ 3 5 9 2 ] : EAL: probe d r i v e r : 8 0 8 6 : 1 5 2 8 rte_ixgbe_pmd ovs−v s w i t c h d [ 3 5 9 2 ] : EAL: PCI memory mapped a t 0 x 7 f 2 1 4 0 0 0 0 0 0 0 ovs−v s w i t c h d [ 3 5 9 2 ] : EAL: PCI memory mapped a t 0 x 7 f 2 1 4 0 2 0 0 0 0 0 ovs−c t l [ 3 5 6 0 ] : EAL: Master l c o r e 0 i s ready ( t i d=f c 6 c b b 0 0 ; c p u s e t = [ 0 ] ) ovs−c t l [ 3 5 6 0 ] : EAL: PCI d e v i c e 0 0 0 0 : 0 4 : 0 0 . 0 on NUMA s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: probe d r i v e r : 8 0 8 6 : 1 5 2 8 rte_ixgbe_pmd ovs−c t l [ 3 5 6 0 ] : EAL: Not managed by a s u p p o r t e d k e r n e l d r i v e r , s k i p p e d ovs−c t l [ 3 5 6 0 ] : EAL: PCI d e v i c e 0 0 0 0 : 0 4 : 0 0 . 1 on NUMA s o c k e t 0 ovs−c t l [ 3 5 6 0 ] : EAL: probe d r i v e r : 8 0 8 6 : 1 5 2 8 rte_ixgbe_pmd ovs−c t l [ 3 5 6 0 ] : EAL: PCI memory mapped a t 0 x 7 f 2 1 4 0 0 0 0 0 0 0 ovs−c t l [ 3 5 6 0 ] : EAL: PCI memory mapped a t 0 x 7 f 2 1 4 0 2 0 0 0 0 0 ovs−v s w i t c h d [ 3 5 9 2 ] : PMD: eth_ixgbe_dev_init ( ) : MAC: 4 , PHY: 3 ovs−v s w i t c h d [ 3 5 9 2 ] : PMD: eth_ixgbe_dev_init ( ) : p o r t 0 vendorID=0x8086 d e v i c e I D =0x1528 ovs−c t l [ 3 5 6 0 ] : PMD: eth_ixgbe_dev_init ( ) : MAC: 4 , PHY: 3 ovs−c t l [ 3 5 6 0 ] : PMD: eth_ixgbe_dev_init ( ) : p o r t 0 vendorID=0x8086 d e v i c e I D=0 x1528 ovs−c t l [ 3 5 6 0 ] : Zone 0 : name: v i r t : 0 x 7 f 2 1 3 f f f d e c 0 , s o c k e t _ i d : 0 , f l a g s : 0 ovs−c t l [ 3 5 6 0 ] : Zone 1 : name: v i r t : 0 x 7 f 2 1 3 f d 7 3 d 4 0 , s o c k e t _ i d : 0 , f l a g s : 0 ovs−c t l [ 3 5 6 0 ] : Zone 2 : name:< rte_eth_dev_data >, phys : 0 x83fd43380 , l e n : 0 x2f700 , v i r t : 0 x 7 f 2 1 3 f d 4 3 3 8 0 , s o c k e t _ i d : 0 , f l a g s : 0 ovs−c t l [ 3 5 6 0 ] : * S t a r t i n g ovs−v s w i t c h d ovs−c t l [ 3 5 6 0 ] : * Ena blin g remote OVSDB managers systemd [ 1 ] : S t a r t e d Open vSwitch I n t e r n a l Unit . systemd [ 1 ] : S t a r t i n g Open vSwitch . . . systemd [ 1 ] : S t a r t e d Open vSwitch . 74 CMD: sudo ovs−v s c t l add−br ovsdpdkbr0 −− s e t b r i d g e ovsdpdkbr0 datapath_type= netdev SYSLOG: 2016−01−22T08 : 5 8 : 5 6 . 3 4 4 Z | 0 0 0 0 8 | memory | INFO| 3 7 2 5 6 kB peak r e s i d e n t s e t s i z e a f t e r 2 4 . 5 s e c o n d s 2016−01−22T08 : 5 8 : 5 6 . 3 4 6 Z | 0 0 0 0 9 | o f p r o t o _ d p i f | INFO | netdev@ovs−netdev : Datapath s u p p o r t s r e c i r c u l a t i o n 2016−01−22T08 : 5 8 : 5 6 . 3 4 6 Z | 0 0 0 1 0 | o f p r o t o _ d p i f | INFO | netdev@ovs−netdev : MPLS l a b e l s t a c k l e n g t h probed a s 3 2016−01−22T08 : 5 8 : 5 6 . 3 4 6 Z | 0 0 0 1 1 | o f p r o t o _ d p i f | INFO | netdev@ovs−netdev : Datapath s u p p o r t s unique f l o w i d s 2016−01−22T08 : 5 8 : 5 6 . 3 4 6 Z | 0 0 0 1 2 | o f p r o t o _ d p i f | INFO | netdev@ovs−netdev : Datapath d o e s not s u p p o r t c t _ s t a t e 2016−01−22T08 : 5 8 : 5 6 . 3 4 6 Z | 0 0 0 1 3 | o f p r o t o _ d p i f | INFO | netdev@ovs−netdev : Datapath d o e s not s u p p o r t ct_zone 2016−01−22T08 : 5 8 : 5 6 . 3 4 6 Z | 0 0 0 1 4 | o f p r o t o _ d p i f | INFO | netdev@ovs−netdev : Datapath d o e s not s u p p o r t ct_mark 2016−01−22T08 : 5 8 : 5 6 . 3 4 6 Z | 0 0 0 1 5 | o f p r o t o _ d p i f | INFO | netdev@ovs−netdev : Datapath d o e s not s u p p o r t c t _ l a b e l 2016−01−22T08 : 5 8 : 5 6 . 3 6 0 Z | 0 0 0 1 6 | b r i d g e | INFO | b r i d g e ovsdpdkbr0 : added i n t e r f a c e ovsdpdkbr0 on p o r t 65534 2016−01−22T08 : 5 8 : 5 6 . 3 6 1 Z | 0 0 0 1 7 | b r i d g e | INFO | b r i d g e ovsdpdkbr0 : u s i n g datapath ID 00005 a4a1ed0a14d 2016−01−22T08 : 5 8 : 5 6 . 3 6 1 Z | 0 0 0 1 8 | connmgr | INFO | ovsdpdkbr0 : added s e r v i c e c o n t r o l l e r ” punix : / var / run / o p e n v s w i t c h / ovsdpdkbr0 . mgmt” OVS−LOG: ovs−v s c t l : ovs | 0 0 0 0 1 | v s c t l | INFO | C a l l e d a s ovs−v s c t l add−br ovsdpdkbr0 −− s e t b r i d g e ovsdpdkbr0 datapath_type=netdev systemd−udevd [ 3 6 0 7 ] : Could not g e n e r a t e p e r s i s t e n t MAC a d d r e s s f o r ovs−netdev : No such f i l e o r d i r e c t o r y k e r n e l : [ 5 0 1 6 5 . 8 8 6 5 5 4 ] d e v i c e ovs−netdev e n t e r e d p r omi s cu ou s mode k e r n e l : [ 5 0 1 6 5 . 9 0 1 2 6 1 ] d e v i c e ovsdpdkbr0 e n t e r e d p r om i s cu o u s mode CMD: sudo ovs−v s c t l add−p o r t ovsdpdkbr0 dpdk0 −− s e t I n t e r f a c e dpdk0 type=dpdk SYSLOG: 2016−01−22T08 : 5 9 : 0 6 . 3 6 9 Z | 0 0 0 1 9 | memory | INFO | peak r e s i d e n t s e t s i z e grew 155% i n l a s t 1 0 . 0 s e c o n d s , from 37256 kB t o 95008 kB 2016−01−22T08 : 5 9 : 0 6 . 3 6 9 Z | 0 0 0 2 0 | memory | INFO | h a n d l e r s : 4 p o r t s : 1 r e v a l i d a t o r s : 2 r u l e s : 5 2016−01−22T08 : 5 9 : 3 0 . 9 8 9 Z | 0 0 0 2 1 | dpdk | INFO | Port 0 : 8 c : dc : d4 : b3 : 6 d : e9 2016−01−22T08 : 5 9 : 3 1 . 5 2 0 Z | 0 0 0 2 2 | dpdk | INFO | Port 0 : 8 c : dc : d4 : b3 : 6 d : e9 2016−01−22T08 : 5 9 : 3 1 . 5 2 1 Z | 0 0 0 2 3 | d p i f _ n e t d e v | INFO | Created 1 pmd t h r e a d s on numa node 0 2016−01−22T08 : 5 9 : 3 1 . 5 2 2 Z | 0 0 0 0 1 | d p i f _ n e t d e v ( pmd16 ) | INFO | Core 0 p r o c e s s i n g p o r t ’ dpdk0 ’ 2016−01−22T08 : 5 9 : 3 1 . 5 2 2 Z | 0 0 0 2 4 | b r i d g e | INFO | b r i d g e ovsdpdkbr0 : added i n t e r f a c e dpdk0 on p o r t 1 2016−01−22T08 : 5 9 : 3 1 . 5 2 2 Z | 0 0 0 2 5 | b r i d g e | INFO | b r i d g e ovsdpdkbr0 : u s i n g datapath 75 ID 00008 cdcd4b36de9 2016−01−22T08 : 5 9 : 3 1 . 5 2 3 Z | 0 0 0 0 2 | d p i f _ n e t d e v ( pmd16 ) | INFO | Core 0 p r o c e s s i n g p o r t ’ dpdk0 ’ OVS−LOG: ovs−v s c t l : ovs | 0 0 0 0 1 | v s c t l | INFO | C a l l e d a s ovs−v s c t l add−p o r t ovsdpdkbr0 dpdk0 −− s e t I n t e r f a c e dpdk0 type=dpdk ovs−v s w i t c h d [ 3 5 9 5 ] : PMD: ixgbe_dev_tx_queue_setup ( ) : sw_ring=0x 7 f 2 1 1 a 7 9 e b c 0 hw_ring=0x 7 f 2 1 1 a 7 a 6 c 0 0 dma_addr=0x81a7a6c00 ovs−v s w i t c h d [ 3 5 9 5 ] : PMD: i x g b e _ s e t _ t x _ f u n c t i o n ( ) : Using s i m p l e tx code path ovs−v s w i t c h d [ 3 5 9 5 ] : PMD: i x g b e _ s e t _ t x _ f u n c t i o n ( ) : Vector tx e n a b l e d . ovs−v s w i t c h d [ 3 5 9 5 ] : PMD: ixgbe_dev_rx_queue_setup ( ) : sw_ring=0x 7 f 2 1 1 a 7 8 a 6 c 0 sw_sc_ring=0x 7 f 2 1 1 a 7 8 6 5 8 0 hw_ring=0x 7 f 2 1 1 a 7 8 e 8 0 0 dma_addr=0x81a78e800 ovs−v s w i t c h d [ 3 5 9 5 ] : PMD: i x g b e _ s e t _ r x _ f u n c t i o n ( ) : Vector rx enabled , p l e a s e make s u r e RX b u r s t s i z e no l e s s than 4 ( p o r t =0) . ovs−v s w i t c h d [ 3 5 9 5 ] : PMD: ixgbe_dev_tx_queue_setup ( ) : sw_ring=0x 7 f 2 1 1 a 7 9 e b c 0 hw_ring=0x 7 f 2 1 1 a 7 a 6 c 0 0 dma_addr=0x81a7a6c00 . . . CMD: sudo ovs−v s c t l add−p o r t ovsdpdkbr0 vhost−u s e r −1 −− s e t I n t e r f a c e vhost− u s e r −1 type=d p d k v h o s t u s e r OVS−LOG: 2016−01−22T09 : 0 0 : 3 5 . 1 4 5 Z | 0 0 0 2 6 | dpdk | INFO | S o c k e t / var / run / o p e n v s w i t c h / vhost− u s e r −1 c r e a t e d f o r vhost−u s e r p o r t vhost−u s e r −1 2016−01−22T09 : 0 0 : 3 5 . 1 4 5 Z | 0 0 0 0 3 | d p i f _ n e t d e v ( pmd16 ) | INFO | Core 0 p r o c e s s i n g p o r t ’ dpdk0 ’ 2016−01−22T09 : 0 0 : 3 5 . 1 4 5 Z | 0 0 0 0 4 | d p i f _ n e t d e v ( pmd16 ) | INFO | Core 0 p r o c e s s i n g p o r t ’ vhost−u s e r −1’ 2016−01−22T09 : 0 0 : 3 5 . 1 4 5 Z | 0 0 0 2 7 | b r i d g e | INFO | b r i d g e ovsdpdkbr0 : added i n t e r f a c e vhost−u s e r −1 on p o r t 2 SYSLOG: ovs−v s c t l : ovs | 0 0 0 0 1 | v s c t l | INFO | C a l l e d a s ovs−v s c t l add−p o r t ovsdpdkbr0 vhost− u s e r −1 −− s e t I n t e r f a c e vhost−u s e r −1 type=d p d k v h o s t u s e r ovs−v s w i t c h d [ 3 5 9 5 ] : VHOST_CONFIG: s o c k e t c r e a t e d , f d : 4 6 ovs−v s w i t c h d [ 3 5 9 5 ] : VHOST_CONFIG: bind t o / var / run / o p e n v s w i t c h / vhost−u s e r −1 E v e n t u a l l y we can s e e t h e p o l l t h r e a d i n top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3595 r o o t 10 −10 4975344 103936 9916 S 1 0 0 . 0 0 . 3 3 3 : 1 3 . 5 6 ovs− v s w i t c h d Download 1.27 Mb. Do'stlaringiz bilan baham: 
|